[BUG] Datapusher broken

[hbenkhay]

Describe the bug

• the datapusher is not working
• Sqlalchemy version is not compatible
• the datapusher fails to connect to ckan on (host='localhost', port=5000)

Docker images affected CKAN 2.9

Logs

• ModuleNotFoundError: No module named 'tzdata'
• Error getting due jobs from job store 'default': columns argument to select() must be a Python list or other iterable
• datapusher | During handling of the above exception, another exception occurred: datapusher | requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /dataset/3d483754-836b-4934-9044-61cd10150fec/resource/fb66fdbf-81a5-472a-8b6b-7e5463aedce4/download/datapusher_test.csv (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb2c8570d60>: Failed to establish a new connection: [Errno 111] Connection refused'))

[hbenkhay]

duplicate of https://github.com/keitaroinc/docker-ckan/issues/111

[stojanovskis1]

@hbenkhay I tried locally with image keitaro/ckan:2.9.9 and keitaro/ckan-datapusher:0.0.17, everything is working as expected, i can't recreate this error, can you share more information?

[hbenkhay]

thank you @stojanovskis1 for looking at this. if you pull the images from DockerHub, I would expect it to work fine, I can also run the services from my docker-compose without an error.

but when I try to build the ckan-datapusher myself from its Dockerfile , I ran into these issues, I think the reason might be in this line

the dockerfile refers to the master branch of another repo, which has been updated since, so even if the Dockerfile stays the same, the python requirements.txt has changed since.

my reason for doing that is to build these images for ARM platform (M1 machines for example), the images availables for ckan and the datapusher are only compatible with AMD.

I already built a CKAN image that support both amd64 and arm64.

so, to reproduce the issue, I would copy the datapusher Dockerfile locally, and build the image without cache.

[stojanovskis1]

@hbenkhay i've changed the requirements.txt on branch change-requirements in datapusher repository. You can just change the branch in the Dockerfile locally to change-requirements instead of master and give it a try to see if the issue is resolved.

[hbenkhay]

I tried that, the container startup is looking good now (no timezone bug), but when I try to upload a dataset, I am getting

datapusher  | Error getting due jobs from job store 'default': columns argument to select() must be a Python list or other iterable
datapusher  | Error getting due jobs from job store 'default': columns argument to select() must be a Python list or other iterable

which I refer to in the issue description above and here , that's a sqlachemy issue (you may want to upgrade that, it worked for me using SQLAlchemy==1.3.19)

[stojanovskis1]

@hbenkhay can you confirm that this issue is resolved with latest image?