search

Sage-Bionetworks / synapsePythonClient

Programmatic interface to Synapse services for Python
https://www.synapse.org
Apache License 2.0
67 stars 68 forks source link

[SYNPY-1445] Raise exception on HTTP error when retrieving file from storage location #1074

Closed BryanFauble closed 7 months ago

BryanFauble commented 7 months ago

Problem:

  1. When retrieving a file from a storage location and we recieve an HTTP error (403 for example) we were failing in an unexpected location: 
    2024-02-26 10:27:30,985 [client:3044 - DEBUG]: 
Retrying download on error: [<class 'KeyError'>] after progressing 0 bytes
Traceback (most recent call last):
File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/client.py", line 3020, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
File "/home/bfauble/.pyenv/versions/3.8.18/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/client.py", line 3092, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/core/multithread_download/download_threads.py", line 304, in download_file
file_size = _get_file_size(url_info.url)
File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/core/multithread_download/download_threads.py", line 222, in _get_file_size
return int(res_get.headers["Content-Length"])
File "/home/bfauble/.local/share/virtualenvs/synapsePythonClient-17ZKsCr2/lib/python3.8/site-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
KeyError: 'content-length

Solution:

  1. Adding in a call to check the HTTP response status code after getting a pre-signed URL.

Testing:

  1. Unit testing
  2. Verified with manual testing that the exception is logged with/without debug level: 
    SynapseHTTPError: 403 Client Error: Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>BSB8MH0NJ4RYZEWP</RequestId><HostId>D3Hd2Kez08WaxZz55Koe5RAk5J06o4I91R8Ql0GlQHoNxEvMDrRQvy+GoLj9rW2h28I1USnuXb4=</HostId></Error>

With debug:

Retrying download on error: [<class 'synapseclient.core.exceptions.SynapseHTTPError'>] after progressing 0 bytes
Traceback (most recent call last):
  File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/client.py", line 3020, in _downloadFileHandle
    downloaded_path = self._download_from_url_multi_threaded(
  File "/home/bfauble/.pyenv/versions/3.8.18/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/client.py", line 3093, in _download_from_url_multi_threaded
    multithread_download.download_file(self, request)
  File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/core/multithread_download/download_threads.py", line 258, in download_file
    downloader.download_file(download_request)
  File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/core/multithread_download/download_threads.py", line 307, in download_file
    file_size = _get_file_size(url_info.url, request.debug)
  File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/core/multithread_download/download_threads.py", line 224, in _get_file_size
    _raise_for_status(res_get, verbose=debug)
  File "/home/bfauble/BryansGreatWorkspace/synapsePythonClient/synapseclient/core/exceptions.py", line 159, in _raise_for_status
    raise SynapseHTTPError(message, response=response)
synapseclient.core.exceptions.SynapseHTTPError: 403 Client Error: Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>J4YKF9Q2TFXN5WGW</RequestId><HostId>6MATb78rqNN/JYdAwQXLeqkeIj7uxGn8oWPUltF4gIWsjYI5xSH8MqgAZHLoMMpsXH4kvjIdM6Y=</HostId></Error>

>>>>>> Request <<<<<<
https://s3.amazonaws.com/psychencode-migration-bucket-bucket-erzvrdscmdvm/3357888/16ffcb11-3f66-47da-a60e-f4ea5130f631/01-052914_AN19442_BA7.bam?response-content-disposition=attachment%3B%20filename%3D%2201-052914_AN19442_BA7.bam%22%3B%20filename%2A%3Dutf-8%27%2701%252D052914%255FAN19442%255FBA7.bam&response-content-type=application%2Foctet-stream&X-Amz-Security-Token=IQoJb3JpZ2luX2VjENr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJGMEQCICfkBzFgvxFTRl7doFNvnCWDjj%2B%2F11hmu0AMYCUFELlOAiB5%2Bn8c3cQtnhUJBuPRg2AVOfkZXgCPWsl2l%2FgyWrLSxCrDBQjD%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAAaDDMyNTU2NTU4NTgzOSIMs%2FL1J9dFybLzzuWfKpcFLTVFWI18akyiO7SwEM8YSIZgMXD4fv1nKfukIcwP8vfHekK1ucf916AGkacT2DPr5Rw30bVT21eSako98AMlOg2wWCbz61Vwwe3kLOHyqd6kX3OvL4ZlGbgknNGDMfAZzEqcqu19%2BkAmKskVE1nRBcXmTeowGKX92ekhno1s5JIt23rqgBnzFrpcm6Su%2BxRcTq4dTjHPwzugWULdZHZGha8pA7%2BPSElrrXIHfhBx3dv9hjI7fnaRx0tTMBpjKQR6owEoTQc0Q37UWsZimPBAZVWXJY19QrNRruISZ%2B9pb6%2BMK3yw4XW3mSx1f0LR4qfBzDvnkHC0ehb8I2lo1uo29xFJ80j68d5nQmWlZOV6SZT9pPc0jhPRfN1Ktj7q7kz7pO9%2BxpkhIgiqWAapWEEtv3hNW6PwXDmMhoLfDR37jhEwH197B29XAEXkWofGde8h1A8rP9bwhdkNlcFEg33Qu26iyY04dzJCDKNZgrT1IIVzAklZ65KuJI6mM24iRjWvzNVVMEE%2FS%2FP%2BhGJJKbV%2BJbj6wWCevto7cX57NVblsRHOavsgz0lbiRfLfMbjG5Aw8RDAsGfMala1FNDBMebNpIewgFwph%2FkAteuglNGnKlAsFmqqrefcHcFdM1MCxhXu2R88cpLyEHvw03CZMQbXZr60rcbEZTH213Y0HhMAMay%2BtPhLVVr7K5S1%2Bq4tTVkU3ELz1GuY65Z2kronmU4disyMnWoKNHRjFBjpWzHJfHdn9VhtBeypnoAdOTFS%2FGCU%2FSNzJ0WgRQQfmqHQnLbBvaOKVzseE38u0oAabwUwGI1BLLNzO2%2Fiowv2FW6saJIlBWvclUbKDnkoZguNZju9lpt6QQEwlD4puiWsJJCb6iw0SPtFJtydMMGS864GOrIBwN65UAsPZsk2xP%2FADvhChBzeeyX0buCG1pbqf5qffqvC1WtbT3i83p4cxt5ajAnjYHNe%2FfcKKHF6dda8ScoIELnxctzeJLzbOrpTuDnQQe6tb5qkYXY%2Ff1S9gUhh%2BdYpWyj3yVXBTl1p8S04G4n3i8JOsp8V2CwWzdF0FDmLKelK3%2FfgiKwMpGQfNFSzqzLXp8UpVsl%2FyU8MOlqcY1NeSkStZLcZqJf07plcHOuC7ZjmPQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240226T173732Z&X-Amz-SignedHeaders=host&X-Amz-Expires=30&X-Amz-Credential=ASIAUXTJYTGX762LXL7E%2F20240226%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1ba7659f97cf4db167aa31161ce450a3e3cde13f1d9b08a6208195c23294be89 GET
>>> Headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
>>> Body: None

>>>>>> Response <<<<<<
<Response [403]>
>>> Headers: {'x-amz-request-id': 'J4YKF9Q2TFXN5WGW', 'x-amz-id-2': '6MATb78rqNN/JYdAwQXLeqkeIj7uxGn8oWPUltF4gIWsjYI5xSH8MqgAZHLoMMpsXH4kvjIdM6Y=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Mon, 26 Feb 2024 17:37:34 GMT', 'Server': 'AmazonS3'}
>>> Body: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>J4YKF9Q2TFXN5WGW</RequestId><HostId>6MATb78rqNN/JYdAwQXLeqkeIj7uxGn8oWPUltF4gIWsjYI5xSH8MqgAZHLoMMpsXH4kvjIdM6Y=</HostId></Error>
sonarcloud[bot] commented 7 months ago

Quality Gate Passed Quality Gate passed

Issues
1 New issue

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud