I’m getting a “Permission Denied” error for 169 out of 1735 files when trying to run the cfht2caom2 workflow on a large-ish set of CFHT SITELLE files.
I’m using netrc for A&A, because “key too small” errors occur with CADC’s certs.
The command line client works ok, so I have authorization to access the files:
root@dbec07f8c03c:/usr/src/app/cfht2caom2/int_test# cadc-data get --netrc ./netrc CFHT 2274220o.fits.fz
root@dbec07f8c03c:/usr/src/app/cfht2caom2/int_test# cadc-data get --fhead --netrc ./netrc CFHT 2274220o.fits.fz
root@dbec07f8c03c:/usr/src/app/cfht2caom2/int_test#
This is the error message:
2020-06-19 18:17:57,853:ERROR:caom2pipe.execute_composable:2267:Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/cadcutils/net/ws.py", line 534, in check_status
response.raise_for_status()
File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/transfer?fhead=True
This is the relevant part of the stack trace:
File "/usr/src/app/caom2tools/caom2utils/caom2utils/fits2caom2.py", line 3622, in get_cadc_headers
client.get_file(archive, file_id, b, fhead=True)
File "/usr/local/lib/python3.8/site-packages/cadcdata/core.py", line 222, in get_file
protocols = self._get_transfer_protocols(archive, file_name,
File "/usr/local/lib/python3.8/site-packages/cadcdata/core.py", line 530, in _get_transfer_protocols
response = self._data_client.post(
File "/usr/local/lib/python3.8/site-packages/cadcutils/net/ws.py", line 294, in post
return self._get_session().post(self._get_url(resource), kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/cadcutils/net/ws.py", line 483, in send
self.check_status(response)
File "/usr/local/lib/python3.8/site-packages/cadcutils/net/ws.py", line 541, in check_status
raise exceptions.ForbiddenException(orig_exception=e)
cadcutils.exceptions.ForbiddenException: Permission Denied (edited)
The subject should be being passed properly, because it's working for the remainder of the 1735 file accesses.
The credentials are the same in both cases - a netrc file.
I’m getting a “Permission Denied” error for 169 out of 1735 files when trying to run the cfht2caom2 workflow on a large-ish set of CFHT SITELLE files.
I’m using netrc for A&A, because “key too small” errors occur with CADC’s certs.
The command line client works ok, so I have authorization to access the files: root@dbec07f8c03c:/usr/src/app/cfht2caom2/int_test# cadc-data get --netrc ./netrc CFHT 2274220o.fits.fz root@dbec07f8c03c:/usr/src/app/cfht2caom2/int_test# cadc-data get --fhead --netrc ./netrc CFHT 2274220o.fits.fz root@dbec07f8c03c:/usr/src/app/cfht2caom2/int_test#
This is the error message: 2020-06-19 18:17:57,853:ERROR:caom2pipe.execute_composable:2267:Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/cadcutils/net/ws.py", line 534, in check_status response.raise_for_status() File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/transfer?fhead=True This is the relevant part of the stack trace: File "/usr/src/app/caom2tools/caom2utils/caom2utils/fits2caom2.py", line 3622, in get_cadc_headers client.get_file(archive, file_id, b, fhead=True) File "/usr/local/lib/python3.8/site-packages/cadcdata/core.py", line 222, in get_file protocols = self._get_transfer_protocols(archive, file_name, File "/usr/local/lib/python3.8/site-packages/cadcdata/core.py", line 530, in _get_transfer_protocols response = self._data_client.post( File "/usr/local/lib/python3.8/site-packages/cadcutils/net/ws.py", line 294, in post return self._get_session().post(self._get_url(resource), kwargs) File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 578, in post return self.request('POST', url, data=data, json=json, kwargs) File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.8/site-packages/cadcutils/net/ws.py", line 483, in send self.check_status(response) File "/usr/local/lib/python3.8/site-packages/cadcutils/net/ws.py", line 541, in check_status raise exceptions.ForbiddenException(orig_exception=e) cadcutils.exceptions.ForbiddenException: Permission Denied (edited)
The subject should be being passed properly, because it's working for the remainder of the 1735 file accesses.
The credentials are the same in both cases - a netrc file.