search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.12k stars 14.3k forks source link

microsoft-psrp securestring jinja2.exceptions.TemplateAssertionError #43992

Open noderat opened 16 hours ago

noderat commented 16 hours ago

Apache Airflow Provider(s)

microsoft-psrp

Versions of Apache Airflow Providers

apache-airflow-providers-microsoft-psrp==2.8.0

Apache Airflow version

2.10.2

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Official Apache Airflow Helm Chart

Deployment details

Using official helm chart and the following slightly extended Dockerfile:

FROM apache/airflow:2.10.2
RUN pip install --no-cache-dir "apache-airflow[microsoft-psrp]==${AIRFLOW_VERSION}" \
    --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.9.txt"

What happened

jinja2.exceptions.TemplateAssertionError: No filter named 'securestring'

It appears the securestring template filter is not working as intended.

Summarized Error:

ERROR - Exception rendering Jinja template for task 'Write-Output', field 'arguments'. Template: ["{{ 'foo' | securestring }}"]
  ( ... lengthy stack trace redacted ... )
jinja2.exceptions.TemplateAssertionError: No filter named 'securestring'.
Full execution log including the redacted stack trace: ``` [2024-11-13, 16:11:36 MST] {taskinstance.py:2865} INFO - Starting attempt 1 of 1 [2024-11-13, 16:11:36 MST] {taskinstance.py:2888} INFO - Executing on 2024-11-13 23:11:35.867453+00:00 [2024-11-13, 16:11:36 MST] {warnings.py:112} WARNING - /home/airflow/.local/lib/python3.12/site-packages/airflow/task/task_runner/standard_task_runner.py:70: DeprecationWarning: This process (pid=14633) is multi-threaded, use of fork() may lead to deadlocks in the child. pid = os.fork() [2024-11-13, 16:11:36 MST] {standard_task_runner.py:72} INFO - Started process 14644 to run task [2024-11-13, 16:11:36 MST] {standard_task_runner.py:104} INFO - Running: ['airflow', 'tasks', 'run', 'psrpoperator_securestring_error_reproduction', 'Write-Output', 'manual__2024-11-13T23:11:35.867453+00:00', '--job-id', '802', '--raw', '--subdir', 'DAGS_FOLDER/psrp_operator_securestring_error_repoduction.py', '--cfg-path', '/tmp/tmpwrdenr76'] [2024-11-13, 16:11:36 MST] {standard_task_runner.py:105} INFO - Job 802: Subtask Write-Output [2024-11-13, 16:11:37 MST] {task_command.py:467} INFO - Running on host experimental-temp-airflow-worker-0.experimental-temp-airflow-worker.experimental-temp-airflow.svc.cluster.local [2024-11-13, 16:11:37 MST] {abstractoperator.py:778} ERROR - Exception rendering Jinja template for task 'Write-Output', field 'arguments'. Template: ["{{ 'foo' | securestring }}"] Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/abstractoperator.py", line 770, in _do_render_template_fields rendered_content = self.render_template( ^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/template/templater.py", line 183, in render_template return [self.render_template(element, context, jinja_env, oids) for element in value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/template/templater.py", line 170, in render_template template = jinja_env.from_string(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 1108, in from_string return cls.from_code(self, self.compile(source), gs, None) ^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 768, in compile self.handle_exception(source=source_hint) File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 939, in handle_exception raise rewrite_traceback_stack(source=source) File "", line 1, in template jinja2.exceptions.TemplateAssertionError: No filter named 'securestring'. [2024-11-13, 16:11:37 MST] {taskinstance.py:3310} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 273, in _run_raw_task TaskInstance._execute_task_with_callbacks( File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 3114, in _execute_task_with_callbacks task_orig = self.render_templates(context=context, jinja_env=jinja_env) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 3533, in render_templates original_task.render_template_fields(context, jinja_env) File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 1419, in render_template_fields self._do_render_template_fields(self, self.template_fields, context, jinja_env, set()) File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/abstractoperator.py", line 770, in _do_render_template_fields rendered_content = self.render_template( ^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/template/templater.py", line 183, in render_template return [self.render_template(element, context, jinja_env, oids) for element in value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/template/templater.py", line 170, in render_template template = jinja_env.from_string(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 1108, in from_string return cls.from_code(self, self.compile(source), gs, None) ^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 768, in compile self.handle_exception(source=source_hint) File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 939, in handle_exception raise rewrite_traceback_stack(source=source) File "", line 1, in template jinja2.exceptions.TemplateAssertionError: No filter named 'securestring'. [2024-11-13, 16:11:37 MST] {taskinstance.py:1225} INFO - Marking task as FAILED. dag_id=psrpoperator_securestring_error_reproduction, task_id=Write-Output, run_id=manual__2024-11-13T23:11:35.867453+00:00, execution_date=20241113T231135, start_date=20241113T231136, end_date=20241113T231137 ``` ### What you think should happen instead The minimal example provided should return through XCom the ToString representation of a powershell SecureString which would be `"System.Security.SecureString"` ### How to reproduce The following DAG can be used to reproduce the issue and does not require a remote connection as the task fails before attempting any connection: ```python import datetime from airflow import DAG from airflow.providers.microsoft.psrp.operators.psrp import PsrpOperator with DAG( dag_id="psrpoperator_securestring_error_reproduction", start_date=None, schedule_interval=None, render_template_as_native_obj=True, ): PsrpOperator( psrp_conn_id = "SomeConnection", cmdlet = "Write-Output", arguments = [ "{{ 'foo' | securestring }}" ], ) ``` ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
boring-cyborg[bot] commented 16 hours ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.