Got this error running the extract_lineage.py DAG:
[2022-06-17, 18:23:41 UTC] {standard_task_runner.py:98} ERROR - Failed to execute job 278 for task send_ol_events (HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /api/v1/lineage (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3435705c90>: Failed to establish a new connection: [Errno 111] Connection refused')); 153)
I'm using port 9000 because I'm on a mac; I specified :9000 in .env under OPENLINEAGE_URL.
In chat, @rossturk said:
hmmmm. so I suspect this has something to do with docker networking
i.e., these two docker-compose deployments are in different virtual networks
I have solved this in the past ... by adding a section to the Airflow docker-compose.yaml that tells it to deploy into the Marquez network
Got this error running the extract_lineage.py DAG:
[2022-06-17, 18:23:41 UTC] {standard_task_runner.py:98} ERROR - Failed to execute job 278 for task send_ol_events (HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /api/v1/lineage (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3435705c90>: Failed to establish a new connection: [Errno 111] Connection refused')); 153)
I'm using port 9000 because I'm on a mac; I specified :9000 in .env under OPENLINEAGE_URL.
In chat, @rossturk said: hmmmm. so I suspect this has something to do with docker networking i.e., these two docker-compose deployments are in different virtual networks
I have solved this in the past ... by adding a section to the Airflow docker-compose.yaml that tells it to deploy into the Marquez network