When running on my example files where it fails to proceed beyond creating the metrics-CHECK_METADATA_FILES.json file, I see text like this in the gapstools-flower-1 log which seems to indicate some sort of error (note this is not a complete log as it was very long):
[2023-02-09 04:14:06,108] {cli_action_loggers.py:105} WARNING - Failed to log action with (psycopg2.errors.UndefinedTable) relation "log" does not exist
LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...
^
[SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (%(dttm)s, %(dag_id)s, %(task_id)s, %(event)s, %(execution_date)s, %(owner)s, %(extra)s) RETURNING log.id]
[parameters: {'dttm': datetime.datetime(2023, 2, 9, 4, 14, 6, 58699, tzinfo=Timezone('UTC')), 'dag_id': None, 'task_id': None, 'event': 'cli_flower', 'execution_date': None, 'owner': 'airflow', 'extra': '{"host_name": "ef46e7cc54f4", "full_command": "[\'/usr/local/airflow/.local/bin/airflow\', \'celery\', \'flower\']"}'}]
(Background on this error at: http://sqlalche.me/e/13/f405)
Warning: entry point could not be loaded. Contact its author for help.
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.9/site-packages/click_plugins/core.py", line 37, in decorator
group.add_command(entry_point.load())
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2457, in load
self.require(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2480, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 788, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (celery 5.1.2 (/usr/local/airflow/.local/lib/python3.9/site-packages), Requirement.parse('celery<5.0.0,>=4.3.0; python_version >= "3.7"'))
[2023-02-09 04:14:20,097] {cli_action_loggers.py:105} WARNING - Failed to log action with (psycopg2.errors.UndefinedTable) relation "log" does not exist
LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...
^
[SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (%(dttm)s, %(dag_id)s, %(task_id)s, %(event)s, %(execution_date)s, %(owner)s, %(extra)s) RETURNING log.id]
[parameters: {'dttm': datetime.datetime(2023, 2, 9, 4, 14, 20, 50820, tzinfo=Timezone('UTC')), 'dag_id': None, 'task_id': None, 'event': 'cli_flower', 'execution_date': None, 'owner': 'airflow', 'extra': '{"host_name": "ef46e7cc54f4", "full_command": "[\'/usr/local/airflow/.local/bin/airflow\', \'celery\', \'flower\']"}'}]
(Background on this error at: http://sqlalche.me/e/13/f405)
Warning: entry point could not be loaded. Contact its author for help.
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.9/site-packages/click_plugins/core.py", line 37, in decorator
group.add_command(entry_point.load())
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2457, in load
self.require(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2480, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 788, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (celery 5.1.2 (/usr/local/airflow/.local/lib/python3.9/site-packages), Requirement.parse('celery<5.0.0,>=4.3.0; python_version >= "3.7"'))
Warning: entry point could not be loaded. Contact its author for help.
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.9/site-packages/click_plugins/core.py", line 37, in decorator
group.add_command(entry_point.load())
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2457, in load
self.require(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2480, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 788, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (celery 5.1.2 (/usr/local/airflow/.local/lib/python3.9/site-packages), Requirement.parse('celery<5.0.0,>=4.3.0; python_version >= "3.7"'))
Warning: entry point could not be loaded. Contact its author for help.
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.9/site-packages/click_plugins/core.py", line 37, in decorator
group.add_command(entry_point.load())
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2457, in load
self.require(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2480, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 788, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (celery 5.1.2 (/usr/local/airflow/.local/lib/python3.9/site-packages), Requirement.parse('celery<5.0.0,>=4.3.0; python_version >= "3.7"'))
Warning: entry point could not be loaded. Contact its author for help.
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.9/site-packages/click_plugins/core.py", line 37, in decorator
group.add_command(entry_point.load())
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2457, in load
self.require(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2480, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/airflow/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 788, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (celery 5.1.2 (/usr/local/airflow/.local/lib/python3.9/site-packages), Requirement.parse('celery<5.0.0,>=4.3.0; python_version >= "3.7"'))
When running on my example files where it fails to proceed beyond creating the
metrics-CHECK_METADATA_FILES.jsonfile, I see text like this in thegapstools-flower-1log which seems to indicate some sort of error (note this is not a complete log as it was very long):