sul-dlss / dlme-airflow

This is a new repository to capture the work related to the DLME ETL Pipeline and establish airflow
Apache License 2.0
1 stars 0 forks source link

Bodleian won't index #102

Closed jacobthill closed 2 years ago

jacobthill commented 2 years ago

Bodleian collections are failing on index: https://airflow.dev.dlmenetwork.org/graph?dag_id=bodleian

[2022-03-02 21:33:22,347] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: bodleian.BODLEIAN_ETL.exploring-egypt_etl.index_bodleian_exploring-egypt 2022-03-02T21:31:28.333387+00:00 [queued]>
[2022-03-02 21:33:22,362] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: bodleian.BODLEIAN_ETL.exploring-egypt_etl.index_bodleian_exploring-egypt 2022-03-02T21:31:28.333387+00:00 [queued]>
[2022-03-02 21:33:22,362] {taskinstance.py:1087} INFO - 
--------------------------------------------------------------------------------
[2022-03-02 21:33:22,362] {taskinstance.py:1088} INFO - Starting attempt 1 of 1
[2022-03-02 21:33:22,362] {taskinstance.py:1089} INFO - 
--------------------------------------------------------------------------------
[2022-03-02 21:33:22,385] {taskinstance.py:1107} INFO - Executing <Task(ECSOperator): BODLEIAN_ETL.exploring-egypt_etl.index_bodleian_exploring-egypt> on 2022-03-02T21:31:28.333387+00:00
[2022-03-02 21:33:22,390] {standard_task_runner.py:52} INFO - Started process 408 to run task
[2022-03-02 21:33:22,394] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'bodleian', 'BODLEIAN_ETL.exploring-egypt_etl.index_bodleian_exploring-egypt', '2022-03-02T21:31:28.333387+00:00', '--job-id', '2050', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/harvest_catalog.py', '--cfg-path', '/tmp/tmpp8hp952c', '--error-file', '/tmp/tmpny6dfxes']
[2022-03-02 21:33:22,396] {standard_task_runner.py:77} INFO - Job 2050: Subtask BODLEIAN_ETL.exploring-egypt_etl.index_bodleian_exploring-egypt
[2022-03-02 21:33:22,492] {logging_mixin.py:104} INFO - Running <TaskInstance: bodleian.BODLEIAN_ETL.exploring-egypt_etl.index_bodleian_exploring-egypt 2022-03-02T21:31:28.333387+00:00 [running]> on host ip-10-0-66-58.us-west-2.compute.internal
[2022-03-02 21:33:22,628] {taskinstance.py:1300} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=jtim@stanford.edu, amcollie@stanford.edu
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=bodleian
AIRFLOW_CTX_TASK_ID=BODLEIAN_ETL.exploring-egypt_etl.index_bodleian_exploring-egypt
AIRFLOW_CTX_EXECUTION_DATE=2022-03-02T21:31:28.333387+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-03-02T21:31:28.333387+00:00
[2022-03-02 21:33:22,628] {ecs.py:207} INFO - Running ECS Task - Task definition: dlme-index-from-s3 - on cluster dlme-dev
[2022-03-02 21:33:22,629] {ecs.py:210} INFO - ECSOperator overrides: {'containerOverrides': [{'name': 'dlme-index-from-s3', 'environment': [{'name': 'S3_FETCH_URL', 'value': 'https://dlme-metadata-dev.s3.us-west-2.amazonaws.com/output/output-bodleian-exploring-egypt.ndjson'}]}]}
[2022-03-02 21:33:22,629] {base_aws.py:368} INFO - Airflow Connection: aws_conn_id=aws_conn
[2022-03-02 21:33:22,646] {base_aws.py:391} WARNING - Unable to use Airflow Connection for credentials.
[2022-03-02 21:33:22,646] {base_aws.py:392} INFO - Fallback on boto3 credential strategy
[2022-03-02 21:33:22,646] {base_aws.py:395} INFO - Creating session using boto3 credential strategy region_name=None
[2022-03-02 21:33:23,263] {ecs.py:264} INFO - ECS Task started: {'tasks': [{'attachments': [{'id': '27a50f56-30c8-45e1-8568-b9c96823a28e', 'type': 'ElasticNetworkInterface', 'status': 'PRECREATED', 'details': [{'name': 'subnetId', 'value': 'subnet-05a755dca83416be5'}]}], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'us-west-2a', 'clusterArn': 'arn:aws:ecs:us-west-2:418214828013:cluster/dlme-dev', 'containers': [{'containerArn': 'arn:aws:ecs:us-west-2:418214828013:container/dlme-dev/9c0810fbcc2f42c99c95f7616cc009b8/42873cdf-cd62-4742-b0e9-5dab3705c13c', 'taskArn': 'arn:aws:ecs:us-west-2:418214828013:task/dlme-dev/9c0810fbcc2f42c99c95f7616cc009b8', 'name': 'dlme-index-from-s3', 'image': 'suldlss/dlme:latest', 'lastStatus': 'PENDING', 'networkInterfaces': [], 'cpu': '0', 'memory': '300'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 3, 2, 21, 33, 23, 231000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING', 'enableExecuteCommand': False, 'group': 'family:dlme-index-from-s3', 'lastStatus': 'PROVISIONING', 'launchType': 'FARGATE', 'memory': '8192', 'overrides': {'containerOverrides': [{'name': 'dlme-index-from-s3', 'environment': [{'name': 'S3_FETCH_URL', 'value': 'https://dlme-metadata-dev.s3.us-west-2.amazonaws.com/output/output-bodleian-exploring-egypt.ndjson'}]}], 'inferenceAcceleratorOverrides': []}, 'platformVersion': '1.4.0', 'platformFamily': 'Linux', 'startedBy': 'airflow', 'tags': [], 'taskArn': 'arn:aws:ecs:us-west-2:418214828013:task/dlme-dev/9c0810fbcc2f42c99c95f7616cc009b8', 'taskDefinitionArn': 'arn:aws:ecs:us-west-2:418214828013:task-definition/dlme-index-from-s3:2', 'version': 1, 'ephemeralStorage': {'sizeInGiB': 20}}], 'failures': [], 'ResponseMetadata': {'RequestId': 'e1d7da67-6ae6-47c9-ab11-0dce5424bd46', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'e1d7da67-6ae6-47c9-ab11-0dce5424bd46', 'content-type': 'application/x-amz-json-1.1', 'content-length': '1525', 'date': 'Wed, 02 Mar 2022 21:33:22 GMT'}, 'RetryAttempts': 0}}
[2022-03-02 21:35:36,008] {ecs.py:319} INFO - ECS Task stopped, check status: {'tasks': [{'attachments': [{'id': '27a50f56-30c8-45e1-8568-b9c96823a28e', 'type': 'ElasticNetworkInterface', 'status': 'DELETED', 'details': [{'name': 'subnetId', 'value': 'subnet-05a755dca83416be5'}, {'name': 'networkInterfaceId', 'value': 'eni-0c670af45d7805d50'}, {'name': 'macAddress', 'value': '06:cc:73:fa:aa:9f'}, {'name': 'privateDnsName', 'value': 'ip-10-0-64-50.us-west-2.compute.internal'}, {'name': 'privateIPv4Address', 'value': '10.0.64.50'}]}], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'us-west-2a', 'clusterArn': 'arn:aws:ecs:us-west-2:418214828013:cluster/dlme-dev', 'connectivity': 'CONNECTED', 'connectivityAt': datetime.datetime(2022, 3, 2, 21, 33, 26, 838000, tzinfo=tzlocal()), 'containers': [{'containerArn': 'arn:aws:ecs:us-west-2:418214828013:container/dlme-dev/9c0810fbcc2f42c99c95f7616cc009b8/42873cdf-cd62-4742-b0e9-5dab3705c13c', 'taskArn': 'arn:aws:ecs:us-west-2:418214828013:task/dlme-dev/9c0810fbcc2f42c99c95f7616cc009b8', 'name': 'dlme-index-from-s3', 'image': 'suldlss/dlme:latest', 'runtimeId': '9c0810fbcc2f42c99c95f7616cc009b8-2516614758', 'lastStatus': 'STOPPED', 'exitCode': 1, 'networkBindings': [], 'networkInterfaces': [{'attachmentId': '27a50f56-30c8-45e1-8568-b9c96823a28e', 'privateIpv4Address': '10.0.64.50'}], 'healthStatus': 'UNKNOWN', 'cpu': '0', 'memory': '300'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 3, 2, 21, 33, 23, 231000, tzinfo=tzlocal()), 'desiredStatus': 'STOPPED', 'enableExecuteCommand': False, 'executionStoppedAt': datetime.datetime(2022, 3, 2, 21, 35, 12, 861000, tzinfo=tzlocal()), 'group': 'family:dlme-index-from-s3', 'healthStatus': 'UNKNOWN', 'lastStatus': 'STOPPED', 'launchType': 'FARGATE', 'memory': '8192', 'overrides': {'containerOverrides': [{'name': 'dlme-index-from-s3', 'environment': [{'name': 'S3_FETCH_URL', 'value': 'https://dlme-metadata-dev.s3.us-west-2.amazonaws.com/output/output-bodleian-exploring-egypt.ndjson'}]}], 'inferenceAcceleratorOverrides': []}, 'platformVersion': '1.4.0', 'platformFamily': 'Linux', 'pullStartedAt': datetime.datetime(2022, 3, 2, 21, 33, 56, 757000, tzinfo=tzlocal()), 'pullStoppedAt': datetime.datetime(2022, 3, 2, 21, 34, 37, 897000, tzinfo=tzlocal()), 'startedAt': datetime.datetime(2022, 3, 2, 21, 35, 7, 190000, tzinfo=tzlocal()), 'startedBy': 'airflow', 'stopCode': 'EssentialContainerExited', 'stoppedAt': datetime.datetime(2022, 3, 2, 21, 35, 35, 708000, tzinfo=tzlocal()), 'stoppedReason': 'Essential container in task exited', 'stoppingAt': datetime.datetime(2022, 3, 2, 21, 35, 22, 885000, tzinfo=tzlocal()), 'tags': [], 'taskArn': 'arn:aws:ecs:us-west-2:418214828013:task/dlme-dev/9c0810fbcc2f42c99c95f7616cc009b8', 'taskDefinitionArn': 'arn:aws:ecs:us-west-2:418214828013:task-definition/dlme-index-from-s3:2', 'version': 6, 'ephemeralStorage': {'sizeInGiB': 20}}], 'failures': [], 'ResponseMetadata': {'RequestId': 'c3e2cc85-d24f-472c-9383-f11881add270', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'c3e2cc85-d24f-472c-9383-f11881add270', 'content-type': 'application/x-amz-json-1.1', 'content-length': '2331', 'date': 'Wed, 02 Mar 2022 21:35:35 GMT'}, 'RetryAttempts': 0}}
[2022-03-02 21:35:36,058] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 222, in execute
    self._check_success_task()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 343, in _check_success_task
    raise AirflowException(f'This task is not in success state {task}')
airflow.exceptions.AirflowException: This task is not in success state {'attachments': [{'id': '27a50f56-30c8-45e1-8568-b9c96823a28e', 'type': 'ElasticNetworkInterface', 'status': 'DELETED', 'details': [{'name': 'subnetId', 'value': 'subnet-05a755dca83416be5'}, {'name': 'networkInterfaceId', 'value': 'eni-0c670af45d7805d50'}, {'name': 'macAddress', 'value': '06:cc:73:fa:aa:9f'}, {'name': 'privateDnsName', 'value': 'ip-10-0-64-50.us-west-2.compute.internal'}, {'name': 'privateIPv4Address', 'value': '10.0.64.50'}]}], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'us-west-2a', 'clusterArn': 'arn:aws:ecs:us-west-2:418214828013:cluster/dlme-dev', 'connectivity': 'CONNECTED', 'connectivityAt': datetime.datetime(2022, 3, 2, 21, 33, 26, 838000, tzinfo=tzlocal()), 'containers': [{'containerArn': 'arn:aws:ecs:us-west-2:418214828013:container/dlme-dev/9c0810fbcc2f42c99c95f7616cc009b8/42873cdf-cd62-4742-b0e9-5dab3705c13c', 'taskArn': 'arn:aws:ecs:us-west-2:418214828013:task/dlme-dev/9c0810fbcc2f42c99c95f7616cc009b8', 'name': 'dlme-index-from-s3', 'image': 'suldlss/dlme:latest', 'runtimeId': '9c0810fbcc2f42c99c95f7616cc009b8-2516614758', 'lastStatus': 'STOPPED', 'exitCode': 1, 'networkBindings': [], 'networkInterfaces': [{'attachmentId': '27a50f56-30c8-45e1-8568-b9c96823a28e', 'privateIpv4Address': '10.0.64.50'}], 'healthStatus': 'UNKNOWN', 'cpu': '0', 'memory': '300'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 3, 2, 21, 33, 23, 231000, tzinfo=tzlocal()), 'desiredStatus': 'STOPPED', 'enableExecuteCommand': False, 'executionStoppedAt': datetime.datetime(2022, 3, 2, 21, 35, 12, 861000, tzinfo=tzlocal()), 'group': 'family:dlme-index-from-s3', 'healthStatus': 'UNKNOWN', 'lastStatus': 'STOPPED', 'launchType': 'FARGATE', 'memory': '8192', 'overrides': {'containerOverrides': [{'name': 'dlme-index-from-s3', 'environment': [{'name': 'S3_FETCH_URL', 'value': 'https://dlme-metadata-dev.s3.us-west-2.amazonaws.com/output/output-bodleian-exploring-egypt.ndjson'}]}], 'inferenceAcceleratorOverrides': []}, 'platformVersion': '1.4.0', 'platformFamily': 'Linux', 'pullStartedAt': datetime.datetime(2022, 3, 2, 21, 33, 56, 757000, tzinfo=tzlocal()), 'pullStoppedAt': datetime.datetime(2022, 3, 2, 21, 34, 37, 897000, tzinfo=tzlocal()), 'startedAt': datetime.datetime(2022, 3, 2, 21, 35, 7, 190000, tzinfo=tzlocal()), 'startedBy': 'airflow', 'stopCode': 'EssentialContainerExited', 'stoppedAt': datetime.datetime(2022, 3, 2, 21, 35, 35, 708000, tzinfo=tzlocal()), 'stoppedReason': 'Essential container in task exited', 'stoppingAt': datetime.datetime(2022, 3, 2, 21, 35, 22, 885000, tzinfo=tzlocal()), 'tags': [], 'taskArn': 'arn:aws:ecs:us-west-2:418214828013:task/dlme-dev/9c0810fbcc2f42c99c95f7616cc009b8', 'taskDefinitionArn': 'arn:aws:ecs:us-west-2:418214828013:task-definition/dlme-index-from-s3:2', 'version': 6, 'ephemeralStorage': {'sizeInGiB': 20}}
[2022-03-02 21:35:36,060] {taskinstance.py:1544} INFO - Marking task as FAILED. dag_id=bodleian, task_id=BODLEIAN_ETL.exploring-egypt_etl.index_bodleian_exploring-egypt, execution_date=20220302T213128, start_date=20220302T213322, end_date=20220302T213536
[2022-03-02 21:35:36,153] {local_task_job.py:149} INFO - Task exited with return code 1

The errors displaying in dev are "File not found:"

Screen Shot 2022-03-02 at 5 09 49 PM
aaron-collier commented 2 years ago

@jacobthill I believe this is now closable, yes?