search

buda-base / ao-workflows

Use DAG platform to define and orchestrate workflows
0 stars 0 forks source link

Don't succeed when aws download fails #11

Closed jimk-bdrc closed 3 months ago

jimk-bdrc commented 4 months ago

A download task had this error:

[2024-03-14T15:41:21.312+0000] {logging_mixin.py:188} INFO - using secrets
[2024-03-14T15:41:21.313+0000] {logging_mixin.py:188} INFO - section='default'   ['default']
[2024-03-14T15:41:21.314+0000] {logging_mixin.py:188} INFO - KeyError: 'region_name'

yet the task succeeded. It should have failed.

jimk-bdrc commented 3 months ago

Extend to cover other AWS failures

[2024-04-12, 08:22:48 EDT] {logging_mixin.py:188} WARNING - /home/***/.local/lib/python3.11/site-packages/***/settings.py:194 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
[2024-04-12, 08:22:48 EDT] {task_command.py:423} INFO - Running <TaskInstance: sqs_scheduled_dag.download_from_messages manual__2024-04-12T12:22:39.930155+00:00 [running]> on host 1e1e0bb0c161
[2024-04-12, 08:22:48 EDT] {taskinstance.py:2513} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='sqs_scheduled_dag' AIRFLOW_CTX_TASK_ID='download_from_messages' AIRFLOW_CTX_EXECUTION_DATE='2024-04-12T12:22:39.930155+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-04-12T12:22:39.930155+00:00'
[2024-04-12, 08:22:48 EDT] {logging_mixin.py:188} INFO - ("download_full_path=PosixPath('/home/***/bdrc/data/Incoming/Archive0/00/W1NLM4700/W1NLM4700.bag.zip') "
 "dfp_str='/home/***/bdrc/data/Incoming/Archive0/00/W1NLM4700/W1NLM4700.bag.zip'")
[2024-04-12, 08:22:48 EDT] {logging_mixin.py:188} INFO - using secrets
[2024-04-12, 08:22:48 EDT] {logging_mixin.py:188} INFO - section='ap_northeast'   ['default', 'ap_northeast']
[2024-04-12, 08:22:52 EDT] {logging_mixin.py:188} INFO - ('Could not retrieve '
 'S3://glacier.staging.nlm.bdrc.org/Archive0/00/W1NLM4700/W1NLM4700.bag.zip:  '
 'An error occurred (InvalidObjectState) when calling the GetObject operation: '
 "The operation is not valid for the object's storage class ")
[2024-04-12, 08:22:52 EDT] {logging_mixin.py:188} INFO - 'No messages'
[2024-04-12, 08:22:52 EDT] {python.py:202} INFO - Done. Returned value was: []
[2024-04-12, 08:22:52 EDT] {taskinstance.py:1149} INFO - Marking task as SUCCESS. dag_id=sqs_scheduled_dag, task_id=download_from_messages, execution_date=20240412T122239, start_date=20240412T122248, end_date=20240412T122252
[2024-04-12, 08:22:52 EDT] {local_task_job_runner.py:234} INFO - Task exited with return code 0
[2024-04-12, 08:22:52 EDT] {taskinstance.py:3312} INFO - 1 downstream tasks scheduled from follow-on schedule check