ckan / datapusher

A standalone web service that pushes data files from a CKAN site resources into its DataStore
GNU Affero General Public License v3.0
77 stars 155 forks source link

Error when uploading data #59

Open EnxEng opened 9 years ago

EnxEng commented 9 years ago

Hi to all. I have the following errors when uploading the file with DataPusher:

Invalid HTTP response: HTTP Error 403: Forbidden

or [u' File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job\n retval = job.func(_job.args, _job.kwargs)\n', u' File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 238, in push_to_datastore\n response = urllib2.urlopen(request, timeout=DOWNLOAD_TIMEOUT)\n', u' File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen\n return _opener.open(url, data, timeout)\n', u' File "/usr/lib/python2.7/urllib2.py", line 400, in open\n response = self._open(req, data)\n', u' File "/usr/lib/python2.7/urllib2.py", line 418, in _open\n \'_open\', req)\n', u' File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain\n result = func(args)\n', u' File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open\n return self.do_open(httplib.HTTPConnection, req)\n', u' File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open\n raise URLError(err)\n', u"URLError(error(111, 'Connection refused'),)"]

or [u' File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job\n retval = job.func(_job.args, _job.kwargs)\n', u' File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 238, in push_to_datastore\n response = urllib2.urlopen(request, timeout=DOWNLOAD_TIMEOUT)\n', u' File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen\n return _opener.open(url, data, timeout)\n' u' File "/usr/lib/python2.7/urllib2.py", line 400, in open\n response = self._open(req, data)\n', u' File "/usr/lib/python2.7/urllib2.py", line 418, in _open\n \'_open\', req)\n', u' File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain\n result = func(args)\n', u' File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open\n return self.do_open(httplib.HTTPConnection, req)\n', u' File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open\n raise URLError(err)\n', u"URLError(gaierror(-3, 'Temporary failure in name resolution'),)"]

or

[u' File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job\n retval = job.func(_job.args, _job.kwargs)\n', u' File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 238, in push_to_datastore\n response = urllib2.urlopen(request, timeout=DOWNLOAD_TIMEOUT)\n', u' File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen\n return _opener.open(url, data, timeout)\n', u' File "/usr/lib/python2.7/urllib2.py", line 400, in open\n response = self._open(req, data)\n', u' File "/usr/lib/python2.7/urllib2.py", line 418, in _open\n \'_open\', req)\n', u' File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain\n result = func(args)\n', u' File "/usr/lib/python2.7/urllib2.py", line 1215, in https_open\n return self.do_open(httplib.HTTPSConnection, req)\n', u' File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open\n raise URLError(err)\n', u"URLError(error(101, 'Network is unreachable'),)"]

it seems to me that the problem is because the server is running behind a proxy.

And perhaps, the opener is not correctly setted, by getting the proxy options from the system, i.e.: proxy = urllib2.ProxyHandler() opener = urllib2.build_opener(proxy)

clementmouchet commented 9 years ago

403 & 503 are typical of a service behind a proxy, you need to specify a no_proxy rule in your network config.

EnxEng commented 9 years ago

Sometimes, you can't fix a no_proxy rule, for a test server.