search

Sage-Bionetworks / synapsePythonClient

Programmatic interface to Synapse services for Python
https://www.synapse.org
Apache License 2.0
67 stars 68 forks source link

KeyError: 'X-Amz-Date' when trying to download HTAN data #1138

Closed NKalavros closed 2 weeks ago

NKalavros commented 3 weeks ago

Hi team,

I've been trying to download some HTAN data, but have been unable to due to a dict KeyError.

Thank you in advance for your help, Nikolas

Operating system

RHEL 8.6

Client version

4.5.1, Python 3.12 (replicated error in Python 3.9)

Description of the problem

Currently trying to download data from HTAN.

import synapseclient
syn = synapseclient.Synapse(debug=True)
syn.login(authToken=os.environ["SYNAPSE_TOKEN"])
syn.get('syn26017536')

Expected behavior

The file being downloaded

Actual behavior

[ERROR] Error occurred while running <coroutine object download_file_entity at 0x1555520efb50> in a sync context. Traceback (most recent call last): File "/gpfs/home/nk4167/miniconda/lib/python3.12/site-packages/synapseclient/core/async_utils.py", line 93, in wrap_async_to_sync return asyncio.run(coroutine) ^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs/home/nk4167/miniconda/lib/python3.12/asyncio/runners.py", line 194, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/gpfs/home/nk4167/miniconda/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs/home/nk4167/miniconda/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/gpfs/home/nk4167/miniconda/lib/python3.12/site-packages/synapseclient/core/download/download_functions.py", line 172, in download_file_entity download_path = await download_by_file_handle( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs/home/nk4167/miniconda/lib/python3.12/site-packages/synapseclient/core/download/download_functions.py", line 515, in download_by_file_handle downloaded_path = await loop.run_in_executor( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs/home/nk4167/miniconda/lib/python3.12/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs/home/nk4167/miniconda/lib/python3.12/site-packages/synapseclient/core/download/download_functions.py", line 517, in lambda: download_from_url( ^^^^^^^^^^^^^^^^^^ File "/gpfs/home/nk4167/miniconda/lib/python3.12/site-packages/synapseclient/core/download/download_functions.py", line 751, in download_from_url ) + PresignedUrlProvider._TIME_BUFFER >= _pre_signed_url_expiration_time( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs/home/nk4167/miniconda/lib/python3.12/site-packages/synapseclient/core/download/download_async.py", line 225, in _pre_signed_url_expiration_time time_made = parsed_query["X-Amz-Date"][0]


KeyError: 'X-Amz-Date'

KeyError: 'X-Amz-Date'

Also happens when downloading from command line.
thomasyu888 commented 3 weeks ago

Hi @NKalavros ,

I see that you're using python 3.12, the current release doesn't support that version of python but that will be coming up soon.

If you install the develop branch of this repo, it should work.

Edit: I see you replicated the error with python 3.9, we will take a look

NKalavros commented 3 weeks ago

Hi @thomasyu888 ,

Thanks for getting back to me so quickly. Yes, and to be more precise:

python --version
Python 3.9.20
[ERROR] Error occurred while running <coroutine object download_file_entity at 0x155551723ac0> in a sync context.
Traceback (most recent call last):
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/async_utils.py", line 93, in wrap_async_to_sync
    return asyncio.run(coroutine)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 172, in download_file_entity
    download_path = await download_by_file_handle(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 515, in download_by_file_handle
    downloaded_path = await loop.run_in_executor(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 517, in <lambda>
    lambda: download_from_url(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 751, in download_from_url
    ) + PresignedUrlProvider._TIME_BUFFER >= _pre_signed_url_expiration_time(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_async.py", line 225, in _pre_signed_url_expiration_time
    time_made = parsed_query["X-Amz-Date"][0]
KeyError: 'X-Amz-Date'

KeyError: 'X-Amz-Date'
thomasyu888 commented 3 weeks ago

Thanks @NKalavros for confirming, could you turn debug to be true and paste the logs?

syn = synapseclient.Synapse(debug=True)
syn.login()
...
thomasyu888 commented 3 weeks ago

note, I can reproduce this bug, this is most likely due to not accounting for downloads from a google bucket, but I will have someone on my team look at this early next week.

NKalavros commented 3 weeks ago

Hi @thomasyu888

Sure thing, here's the code I used

import os
import synapseclient
syn = synapseclient.Synapse(debug=True)
syn.login(authToken=os.environ["SYNAPSE_TOKEN"])
syn.get('syn26017536')

Here's the debug output:

2024-09-27 15:35:22,737 [client:6075 - DEBUG]: Sending post request to /entity/syn26017536/bundle2
Downloading from https://storage.googleapis.com/htan-dcc-htapp/visium_level_3_lung/V10U24-037_A_filtered_barcodes.tsv.gz?GoogleAccessId=synapse-svc-prod@uplifted-crow-246820.iam.gserviceaccount.com&Expires=1727465752&Signature=EZpFuxaFzqIF64yyZadXj44GKFY%2FKQXa1LMKfiY29RPDEf8vxHK7FNWPKA9HHQqsnNm4LzUWHkiNVydF6pjynTarJFAcxpigPKaEgLmna1oyMOTNV%2BxTF9AgyjaBMIXl0M29lGYvgALOqyEEhB3Q1MvBVMS66x203j7o7hj9sN%2B8kxMg63bSmFA%2Fu9NN8UoPhvhcT89S10YEI4WVaos4UQHzUSaVQKNwQ5JEnhldXOtTKh6gmLeaj8yXLWylqx00teNQcCzCwo8nj2jyyIi1Fo%2BDnj%2BAsxhRA8btbOU9cBqCOlXPAdZZXs1XcUXKJUP2J2tCfPiyoLqy1r0gBGfVAg%3D%3D&response-content-disposition=attachment%3B%20filename%3D%22V10U24-037_A_filtered_barcodes.tsv.gz%22%3B%20filename%2A%3Dutf-8%27%27V10U24%252D037%255FA%255Ffiltered%255Fbarcodes.tsv.gz&response-content-type=text%2Ftab-separated-values to /gpfs/home/nk4167/.synapseCache/136/79833136/V10U24-037_A_filtered_barcodes.tsv.gz

Retrying download on error: [<class 'KeyError'>] after progressing 0 bytes
Traceback (most recent call last):
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 515, in download_by_file_handle
    downloaded_path = await loop.run_in_executor(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 517, in <lambda>
    lambda: download_from_url(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 751, in download_from_url
    ) + PresignedUrlProvider._TIME_BUFFER >= _pre_signed_url_expiration_time(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_async.py", line 225, in _pre_signed_url_expiration_time
    time_made = parsed_query["X-Amz-Date"][0]
KeyError: 'X-Amz-Date'
Downloading from https://storage.googleapis.com/htan-dcc-htapp/visium_level_3_lung/V10U24-037_A_filtered_barcodes.tsv.gz?GoogleAccessId=synapse-svc-prod@uplifted-crow-246820.iam.gserviceaccount.com&Expires=1727465752&Signature=EZpFuxaFzqIF64yyZadXj44GKFY%2FKQXa1LMKfiY29RPDEf8vxHK7FNWPKA9HHQqsnNm4LzUWHkiNVydF6pjynTarJFAcxpigPKaEgLmna1oyMOTNV%2BxTF9AgyjaBMIXl0M29lGYvgALOqyEEhB3Q1MvBVMS66x203j7o7hj9sN%2B8kxMg63bSmFA%2Fu9NN8UoPhvhcT89S10YEI4WVaos4UQHzUSaVQKNwQ5JEnhldXOtTKh6gmLeaj8yXLWylqx00teNQcCzCwo8nj2jyyIi1Fo%2BDnj%2BAsxhRA8btbOU9cBqCOlXPAdZZXs1XcUXKJUP2J2tCfPiyoLqy1r0gBGfVAg%3D%3D&response-content-disposition=attachment%3B%20filename%3D%22V10U24-037_A_filtered_barcodes.tsv.gz%22%3B%20filename%2A%3Dutf-8%27%27V10U24%252D037%255FA%255Ffiltered%255Fbarcodes.tsv.gz&response-content-type=text%2Ftab-separated-values to /gpfs/home/nk4167/.synapseCache/136/79833136/V10U24-037_A_filtered_barcodes.tsv.gz

Retrying download on error: [<class 'KeyError'>] after progressing 0 bytes
Traceback (most recent call last):
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 515, in download_by_file_handle
    downloaded_path = await loop.run_in_executor(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 517, in <lambda>
    lambda: download_from_url(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 751, in download_from_url
    ) + PresignedUrlProvider._TIME_BUFFER >= _pre_signed_url_expiration_time(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_async.py", line 225, in _pre_signed_url_expiration_time
    time_made = parsed_query["X-Amz-Date"][0]
KeyError: 'X-Amz-Date'
Downloading from https://storage.googleapis.com/htan-dcc-htapp/visium_level_3_lung/V10U24-037_A_filtered_barcodes.tsv.gz?GoogleAccessId=synapse-svc-prod@uplifted-crow-246820.iam.gserviceaccount.com&Expires=1727465753&Signature=iuJuF3ZYNOo731chvHhcPN%2BDDj8imuTFjMUKCJVSuEiRfHoCmQufjTYI6vyUi6%2BqsFoyKB4VVgctr0ZhFIo5q4dhrsrU1b7JqisNETxf%2F20dKU06gWOYD25bF39nXxNS8SmFOLiqk7EjJjVF3sHs%2BPTvHhaZyIFHWaX791IOn7uwFGKpa4jArvpZGEWoezgJgajSwWQ%2BlwEkPKX%2B1Iaz3I%2BqspUVk8Bgr%2Bi5PqmyfcvOQRQ%2Fd80cxgj7EOJ%2F22%2FaeLdFtQOHlu9Urzs669z92vTT5XuEDiLRbUPCn7S1NwLGIOAFLAHKqTnZrJ5U%2Fa7pHxwc9ayYAzP9gxEsDZi2vg%3D%3D&response-content-disposition=attachment%3B%20filename%3D%22V10U24-037_A_filtered_barcodes.tsv.gz%22%3B%20filename%2A%3Dutf-8%27%27V10U24%252D037%255FA%255Ffiltered%255Fbarcodes.tsv.gz&response-content-type=text%2Ftab-separated-values to /gpfs/home/nk4167/.synapseCache/136/79833136/V10U24-037_A_filtered_barcodes.tsv.gz

Retrying download on error: [<class 'KeyError'>] after progressing 0 bytes
Traceback (most recent call last):
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 515, in download_by_file_handle
    downloaded_path = await loop.run_in_executor(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 517, in <lambda>
    lambda: download_from_url(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 751, in download_from_url
    ) + PresignedUrlProvider._TIME_BUFFER >= _pre_signed_url_expiration_time(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_async.py", line 225, in _pre_signed_url_expiration_time
    time_made = parsed_query["X-Amz-Date"][0]
KeyError: 'X-Amz-Date'
Downloading from https://storage.googleapis.com/htan-dcc-htapp/visium_level_3_lung/V10U24-037_A_filtered_barcodes.tsv.gz?GoogleAccessId=synapse-svc-prod@uplifted-crow-246820.iam.gserviceaccount.com&Expires=1727465753&Signature=iuJuF3ZYNOo731chvHhcPN%2BDDj8imuTFjMUKCJVSuEiRfHoCmQufjTYI6vyUi6%2BqsFoyKB4VVgctr0ZhFIo5q4dhrsrU1b7JqisNETxf%2F20dKU06gWOYD25bF39nXxNS8SmFOLiqk7EjJjVF3sHs%2BPTvHhaZyIFHWaX791IOn7uwFGKpa4jArvpZGEWoezgJgajSwWQ%2BlwEkPKX%2B1Iaz3I%2BqspUVk8Bgr%2Bi5PqmyfcvOQRQ%2Fd80cxgj7EOJ%2F22%2FaeLdFtQOHlu9Urzs669z92vTT5XuEDiLRbUPCn7S1NwLGIOAFLAHKqTnZrJ5U%2Fa7pHxwc9ayYAzP9gxEsDZi2vg%3D%3D&response-content-disposition=attachment%3B%20filename%3D%22V10U24-037_A_filtered_barcodes.tsv.gz%22%3B%20filename%2A%3Dutf-8%27%27V10U24%252D037%255FA%255Ffiltered%255Fbarcodes.tsv.gz&response-content-type=text%2Ftab-separated-values to /gpfs/home/nk4167/.synapseCache/136/79833136/V10U24-037_A_filtered_barcodes.tsv.gz

Retrying download on error: [<class 'KeyError'>] after progressing 0 bytes
Traceback (most recent call last):
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 515, in download_by_file_handle
    downloaded_path = await loop.run_in_executor(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 517, in <lambda>
    lambda: download_from_url(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 751, in download_from_url
    ) + PresignedUrlProvider._TIME_BUFFER >= _pre_signed_url_expiration_time(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_async.py", line 225, in _pre_signed_url_expiration_time
    time_made = parsed_query["X-Amz-Date"][0]
KeyError: 'X-Amz-Date'
Downloading from https://storage.googleapis.com/htan-dcc-htapp/visium_level_3_lung/V10U24-037_A_filtered_barcodes.tsv.gz?GoogleAccessId=synapse-svc-prod@uplifted-crow-246820.iam.gserviceaccount.com&Expires=1727465753&Signature=iuJuF3ZYNOo731chvHhcPN%2BDDj8imuTFjMUKCJVSuEiRfHoCmQufjTYI6vyUi6%2BqsFoyKB4VVgctr0ZhFIo5q4dhrsrU1b7JqisNETxf%2F20dKU06gWOYD25bF39nXxNS8SmFOLiqk7EjJjVF3sHs%2BPTvHhaZyIFHWaX791IOn7uwFGKpa4jArvpZGEWoezgJgajSwWQ%2BlwEkPKX%2B1Iaz3I%2BqspUVk8Bgr%2Bi5PqmyfcvOQRQ%2Fd80cxgj7EOJ%2F22%2FaeLdFtQOHlu9Urzs669z92vTT5XuEDiLRbUPCn7S1NwLGIOAFLAHKqTnZrJ5U%2Fa7pHxwc9ayYAzP9gxEsDZi2vg%3D%3D&response-content-disposition=attachment%3B%20filename%3D%22V10U24-037_A_filtered_barcodes.tsv.gz%22%3B%20filename%2A%3Dutf-8%27%27V10U24%252D037%255FA%255Ffiltered%255Fbarcodes.tsv.gz&response-content-type=text%2Ftab-separated-values to /gpfs/home/nk4167/.synapseCache/136/79833136/V10U24-037_A_filtered_barcodes.tsv.gz

Retrying download on error: [<class 'KeyError'>] after progressing 0 bytes
Traceback (most recent call last):
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 515, in download_by_file_handle
    downloaded_path = await loop.run_in_executor(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 517, in <lambda>
    lambda: download_from_url(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 751, in download_from_url
    ) + PresignedUrlProvider._TIME_BUFFER >= _pre_signed_url_expiration_time(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_async.py", line 225, in _pre_signed_url_expiration_time
    time_made = parsed_query["X-Amz-Date"][0]
KeyError: 'X-Amz-Date'
Downloading files:   0%|                                                                                                                                                             | 0.00/1.00 [00:00<?, ?B/s]
2024-09-27 15:35:23,197 [async_utils:96 - ERROR]: Error occurred while running <coroutine object download_file_entity at 0x155550b23940> in a sync context.
Traceback (most recent call last):
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/async_utils.py", line 93, in wrap_async_to_sync
    return asyncio.run(coroutine)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 172, in download_file_entity
    download_path = await download_by_file_handle(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 515, in download_by_file_handle
    downloaded_path = await loop.run_in_executor(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 517, in <lambda>
    lambda: download_from_url(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py", line 751, in download_from_url
    ) + PresignedUrlProvider._TIME_BUFFER >= _pre_signed_url_expiration_time(
  File "/gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_async.py", line 225, in _pre_signed_url_expiration_time
    time_made = parsed_query["X-Amz-Date"][0]
KeyError: 'X-Amz-Date'

Here's also the traceback:

KeyError                                  Traceback (most recent call last)
Cell In[4], line 1
----> 1 syn.get('syn26017536')

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/client.py:1336, in Synapse.get(self, entity, **kwargs)
   1331 # Check and warn for unmet access requirements
   1332 self._check_entity_restrictions(
   1333     bundle, entity, kwargs.get("downloadFile", True)
   1334 )
-> 1336 return_data = self._getWithEntityBundle(
   1337     entityBundle=bundle, entity=entity, **kwargs
   1338 )
   1339 trace.get_current_span().set_attributes(
   1340     {
   1341         "synapse.id": return_data.get("id", ""),
   1342         "synapse.concrete_type": return_data.get("concreteType", ""),
   1343     }
   1344 )
   1345 return return_data

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/client.py:1523, in Synapse._getWithEntityBundle(self, entityBundle, entity, **kwargs)
   1521 if downloadFile:
   1522     if file_handle:
-> 1523         wrap_async_to_sync(
   1524             coroutine=download_file_entity(
   1525                 download_location=downloadLocation,
   1526                 entity=entity,
   1527                 if_collision=ifcollision,
   1528                 submission=submission,
   1529                 synapse_client=self,
   1530             ),
   1531             syn=self,
   1532         )
   1533     else:  # no filehandle means that we do not have DOWNLOAD permission
   1534         warning_message = (
   1535             "WARNING: You have READ permission on this file entity but not DOWNLOAD "
   1536             "permission. The file has NOT been downloaded."
   1537         )

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/async_utils.py:99, in wrap_async_to_sync(coroutine, syn)
     95 except Exception as ex:
     96     syn.logger.exception(
     97         f"Error occurred while running {coroutine} in a sync context."
     98     )
---> 99     raise ex

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/async_utils.py:93, in wrap_async_to_sync(coroutine, syn)
     91         return loop.run_until_complete(coroutine)
     92     else:
---> 93         return asyncio.run(coroutine)
     95 except Exception as ex:
     96     syn.logger.exception(
     97         f"Error occurred while running {coroutine} in a sync context."
     98     )

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/asyncio/runners.py:44, in run(main, debug)
     42     if debug is not None:
     43         loop.set_debug(debug)
---> 44     return loop.run_until_complete(main)
     45 finally:
     46     try:

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/asyncio/base_events.py:647, in BaseEventLoop.run_until_complete(self, future)
    644 if not future.done():
    645     raise RuntimeError('Event loop stopped before Future completed.')
--> 647 return future.result()

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py:172, in download_file_entity(download_location, entity, if_collision, submission, synapse_client)
    168 # reassign downloadPath because if url points to local file (e.g. file://~/someLocalFile.txt)
    169 # it won't be "downloaded" and, instead, downloadPath will just point to '~/someLocalFile.txt'
    170 # _downloadFileHandle may also return None to indicate that the download failed
    171 with logging_redirect_tqdm(loggers=[client.logger]):
--> 172     download_path = await download_by_file_handle(
    173         file_handle_id=entity.dataFileHandleId,
    174         synapse_id=object_id,
    175         entity_type=object_type,
    176         destination=download_path,
    177         synapse_client=client,
    178     )
    180 if download_path is None or not os.path.exists(download_path):
    181     return

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py:515, in download_by_file_handle(file_handle_id, synapse_id, entity_type, destination, retries, synapse_client)
    510     loop = asyncio.get_running_loop()
    511     progress_bar = get_or_create_download_progress_bar(
    512         file_size=1, postfix=synapse_id, synapse_client=syn
    513     )
--> 515     downloaded_path = await loop.run_in_executor(
    516         syn._get_thread_pool_executor(asyncio_event_loop=loop),
    517         lambda: download_from_url(
    518             url=file_handle_result["preSignedURL"],
    519             destination=destination,
    520             entity_id=synapse_id,
    521             file_handle_associate_type=entity_type,
    522             file_handle_id=file_handle["id"],
    523             expected_md5=file_handle.get("contentMd5"),
    524             progress_bar=progress_bar,
    525             synapse_client=syn,
    526         ),
    527     )
    529 syn.logger.info(f"Downloaded {synapse_id} to {downloaded_path}")
    530 syn.cache.add(
    531     file_handle["id"], downloaded_path, file_handle.get("contentMd5", None)
    532 )

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/concurrent/futures/thread.py:58, in _WorkItem.run(self)
     55     return
     57 try:
---> 58     result = self.fn(*self.args, **self.kwargs)
     59 except BaseException as exc:
     60     self.future.set_exception(exc)

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py:517, in download_by_file_handle.<locals>.<lambda>()
    510     loop = asyncio.get_running_loop()
    511     progress_bar = get_or_create_download_progress_bar(
    512         file_size=1, postfix=synapse_id, synapse_client=syn
    513     )
    515     downloaded_path = await loop.run_in_executor(
    516         syn._get_thread_pool_executor(asyncio_event_loop=loop),
--> 517         lambda: download_from_url(
    518             url=file_handle_result["preSignedURL"],
    519             destination=destination,
    520             entity_id=synapse_id,
    521             file_handle_associate_type=entity_type,
    522             file_handle_id=file_handle["id"],
    523             expected_md5=file_handle.get("contentMd5"),
    524             progress_bar=progress_bar,
    525             synapse_client=syn,
    526         ),
    527     )
    529 syn.logger.info(f"Downloaded {synapse_id} to {downloaded_path}")
    530 syn.cache.add(
    531     file_handle["id"], downloaded_path, file_handle.get("contentMd5", None)
    532 )

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_functions.py:751, in download_from_url(url, destination, entity_id, file_handle_associate_type, file_handle_id, expected_md5, progress_bar, synapse_client)
    747 url_is_expired = False
    748 if url_has_expiration:
    749     url_is_expired = datetime.datetime.now(
    750         tz=datetime.timezone.utc
--> 751     ) + PresignedUrlProvider._TIME_BUFFER >= _pre_signed_url_expiration_time(
    752         url
    753     )
    754 if url_is_expired:
    755     response = get_file_handle_for_download(
    756         file_handle_id=file_handle_id,
    757         synapse_id=entity_id,
    758         entity_type=file_handle_associate_type,
    759         synapse_client=client,
    760     )

File /gpfs/scratch/nk4167/envs/instanseg/lib/python3.9/site-packages/synapseclient/core/download/download_async.py:225, in _pre_signed_url_expiration_time(url)
    215 """
    216 Returns time at which a presigned url will expire
    217 
   (...)
    222     A datetime in UTC of when the url will expire
    223 """
    224 parsed_query = parse_qs(urlparse(url).query)
--> 225 time_made = parsed_query["X-Amz-Date"][0]
    226 time_made_datetime = datetime.datetime.strptime(time_made, ISO_AWS_STR_FORMAT)
    227 expires = parsed_query["X-Amz-Expires"][0]

KeyError: 'X-Amz-Date'

This is the case for multiple HTAN IDs.

Best, Nikolas

thomasyu888 commented 3 weeks ago

@NKalavros , thanks - could you try installing version 4.4 of the python client to unblock your work. I tested this. We will make sure to fix in upcoming releases to ensure we continue supporting google buckets

In [4]: syn = synapseclient.login()

UPGRADE AVAILABLE

A more recent version of the Synapse Client (4.5.1) is available. Your version (4.4.0) can be upgraded by typing:
    pip install --upgrade synapseclient

Python Synapse Client version 4.5.1 release notes

https://python-docs.synapse.org/news/

Welcome, Thomas Yu!

In [5]: syn.get('syn26017536')
Downloading files: 100%|█████████████████████████████████████████████████████████████████████████| 10.1k/10.1k [00:00<00:00, 37.9kB/s, syn26017536]
NKalavros commented 3 weeks ago

Hi @thomasyu888

Works! Thanks a bunch, feel free to close this or keep it open 'till next release.

Best, Nikolas

thomasyu888 commented 3 weeks ago

Thanks @NKalavros for taking the time to file this ticket! Glad it worked!

I will leave it open for now, but we may revert the code that introduced this bug and re-design that portion to account for google buckets.

thomasyu888 commented 1 week ago

Just to loop back around @NKalavros, v4.6.0 has the fix!