ckan / ckanext-xloader

Express Loader - quickly load data into DataStore. A replacement for DataPusher.
GNU Affero General Public License v3.0
43 stars 49 forks source link

JobError: Connection timed out after 30s #119

Open clauded opened 3 years ago

clauded commented 3 years ago

I'm getting connection time outs on some site. Would it be possible to define a new ckan.ini variable so the connection time can be increased?

URL time out after 30s
2020-09-25 14:53:43,710 WARNI [fd147473-1c82-4ad8-9f36-6b1706466f99] URL time out after 30s
2020-09-25 14:53:43,741 ERROR [ckanext.xloader.jobs] xloader error: Connection timed out after 30s, Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckanext-xloader/ckanext/xloader/jobs.py", line 73, in xloader_data_into_datastore
    xloader_data_into_datastore_(input, job_dict)
  File "/usr/lib/ckan/default/src/ckanext-xloader/ckanext/xloader/jobs.py", line 223, in xloader_data_into_datastore_
    DOWNLOAD_TIMEOUT))
JobError: Connection timed out after 30s
Ukarus commented 3 months ago

This might be really late, but in case someone experience a similar problem in the future this workaround could be useful.

I was experiencing a similar problem in which I was not able to upload resources to datastore because xloader was getting connection timeout errors when trying to download the resource to upload it to postgres. The production server in which Ckan was operating was behind a private network that was not allowing the server to ping to his own hostname, so a solution I found was to edit the /etc/hosts file and map the hostname to localhost in this way:

127.0.0.1 myhost.com

I hope this turns out to be useful to someone who experiences a similar issue