andalexo
commented
1 year ago Thank you for your feedback !
A change on the DAG generation utility caused old DAGs to fail on this isfile assertion.
The change has been applied on the Echelon* models and is currently applied on the other scenarios (DAGs) as well.
I ran the simulation, Echelon-v2-test-dimitra, for Echelon v2 using the data (https://konnectasystems.sharepoint.com/:f:/r/sites/LEAD/Shared%20Documents/LEAD%20Platform%20Feedback/data/echelon/v2?csf=1&web=1&e=xiVGV4) and received the following error: Error during the extraction of model virtual variables. Please check model virtual variables configuration.
bf048245faa1 *** Reading local file: /opt/airflow/logs/dag_id=echelon_v2_u1/run_id=manual__2023-03-21T11:39:18.580143+00:00/task_id=echelon_v2_v2_u1_3_3_post/attempt=1.log [2023-03-21 11:39:27,139] {taskinstance.py:1159} INFO - Dependencies all met for <TaskInstance: echelon_v2_u1.echelon_v2_v2_u1_3_3_post manual2023-03-21T11:39:18.580143+00:00 [queued]> [2023-03-21 11:39:27,144] {taskinstance.py:1159} INFO - Dependencies all met for <TaskInstance: echelon_v2_u1.echelon_v2_v2_u1_3_3_post manual2023-03-21T11:39:18.580143+00:00 [queued]> [2023-03-21 11:39:27,144] {taskinstance.py:1356} INFO -
[2023-03-21 11:39:27,144] {taskinstance.py:1357} INFO - Starting attempt 1 of 1 [2023-03-21 11:39:27,144] {taskinstance.py:1358} INFO -
[2023-03-21 11:39:27,151] {taskinstance.py:1377} INFO - Executing <Task(_PythonDecoratedOperator): echelon_v2_v2_u1_3_3_post> on 2023-03-21 11:39:18.580143+00:00 [2023-03-21 11:39:27,157] {standard_task_runner.py:52} INFO - Started process 592788 to run task [2023-03-21 11:39:27,159] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'echelon_v2_u1', 'echelon_v2_v2_u1_3_3_post', 'manual2023-03-21T11:39:18.580143+00:00', '--job-id', '1029', '--raw', '--subdir', 'DAGS_FOLDER/echelon_v2_u1.py', '--cfg-path', '/tmp/tmpu0uppul5', '--error-file', '/tmp/tmpaeqi9rvi'] [2023-03-21 11:39:27,160] {standard_task_runner.py:80} INFO - Job 1029: Subtask echelon_v2_v2_u1_3_3_post [2023-03-21 11:39:27,191] {task_command.py:369} INFO - Running <TaskInstance: echelon_v2_u1.echelon_v2_v2_u1_3_3_post manual2023-03-21T11:39:18.580143+00:00 [running]> on host bf048245faa1 [2023-03-21 11:39:27,231] {taskinstance.py:1569} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=echelon_v2_u1 AIRFLOW_CTX_TASK_ID=echelon_v2_v2_u1_3_3_post AIRFLOW_CTX_EXECUTION_DATE=2023-03-21T11:39:18.580143+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=manual__2023-03-21T11:39:18.580143+00:00 [2023-03-21 11:39:27,237] {base.py:68} INFO - Using connection ID 'api_db' for task execution. [2023-03-21 11:39:27,239] {taskinstance.py:1889} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.9/site-packages/airflow/decorators/base.py", line 179, in execute return_value = super().execute(context) File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute return_value = self.execute_callable() File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/opt/airflow/dags/echelon_v2_u1.py", line 335, in echelon_v2_v2_u1_3_3_post assert isfile(filename) AssertionError [2023-03-21 11:39:27,245] {taskinstance.py:1395} INFO - Marking task as FAILED. dag_id=echelon_v2_u1, task_id=echelon_v2_v2_u1_3_3_post, execution_date=20230321T113918, start_date=20230321T113927, end_date=20230321T113927 [2023-03-21 11:39:27,251] {standard_task_runner.py:92} ERROR - Failed to execute job 1029 for task echelon_v2_v2_u1_3_3_post (; 592788) [2023-03-21 11:39:27,291] {local_task_job.py:156} INFO - Task exited with return code 1 [2023-03-21 11:39:27,317] {local_task_job.py:273} INFO - 1 downstream tasks scheduled from follow-on schedule check