search

navis-org / navis-flybrains

Transforms for navis that enable mapping between different Drosophila template brains.
GNU General Public License v3.0
7 stars 2 forks source link

failed to download h5 bridging transforms generated by the Saalfeld lab #3

Closed tedwu-neuro closed 3 years ago

tedwu-neuro commented 3 years ago

Hi,

When I tried to use the code (flybrains.download_saalfeldlab_transforms()), I have the error below:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-25-205a66d97e28> in <module>
----> 1 flybrains.download_saalfeldlab_transforms()

~\Anaconda3\lib\site-packages\flybrains\download.py in download_saalfeldlab_transforms(data_home, skip_existing)
    233         if skip_existing and os.path.exists(dst):
    234             continue
--> 235         _ = download_from_url(url, dst)
    236 
    237 

~\Anaconda3\lib\site-packages\flybrains\download.py in download_from_url(url, dst, resume)
    255 
    256     """
--> 257     file_size = int(requests.head(url, allow_redirects=True).headers["Content-Length"])
    258     if os.path.exists(dst) and resume:
    259         first_byte = os.path.getsize(dst)

~\Anaconda3\lib\site-packages\requests\structures.py in __getitem__(self, key)
     52 
     53     def __getitem__(self, key):
---> 54         return self._store[key.lower()][1]
     55 
     56     def __delitem__(self, key):

KeyError: 'content-length'
--------------------------------

May I how I can solve the issue?

Thanks a lot!

Best, Yifan

schlegelp commented 3 years ago

Hi. This is not an issue on your end. Instead the issue appears to be that the HEAD request to fetch the file size fails. Here is the curl equivalent for the JRC2018F_FAFB.h5 transform:

$ $ curl -L --head https://ndownloader.figshare.com/files/14362754?private_link=3a8b1d84c5e197edc97c 
HTTP/1.1 302 Found
Server: nginx
Date: Sun, 01 Aug 2021 11:48:52 GMT
Content-Type: application/octet-stream
Content-Length: 0
Connection: keep-alive
X-Filename: JRC2018F_FAFB.h5
X-Storage-Protocol: https
Location: https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/14362754/JRC2018F_FAFB.h5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=b3849f006c169fdb24a3fbd45457b1e32c08b268b7252fee2bbb445f7d1b9ab9&X-Amz-Date=20210801T114852Z&X-Amz-Credential=AKIAIYCQYOYV5JSSROOA/20210801/eu-west-1/s3/aws4_request
X-Storage-Host: s3-eu-west-1.amazonaws.com
X-Storage-File: 14362754/JRC2018F_FAFB.h5
X-Storage-Bucket: pfigshare-u-files
Content-Disposition: attachment;filename=JRC2018F_FAFB.h5
Cache-Control: no-cache, no-store
Set-Cookie: fig_tracker_client=d3d71476-932a-4788-a9d7-45744ef63973; Max-Age=31536000; Path=/; expires=Mon, 01-Aug-2022 11:48:52 GMT; secure; HttpOnly
X-Robots-Tag: noindex
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubDomains;
Cache-Control: public, must-revalidate, proxy-revalidate
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET, OPTIONS
Access-Control-Allow-Headers: Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization,Range
Access-Control-Expose-Headers: Location,Accept-Ranges,Content-Encoding,Content-Length,Content-Range

HTTP/1.1 403 Forbidden
x-amz-request-id: 0FS8G6DPVJD9H447
x-amz-id-2: c63ugheKNdLMZtDvy1Pwt+dSZ9t5RhKKGje+/68VPNa8/DLV8f0gxhTZLPpFuY6zRegLZU6zogE=
Content-Type: application/xml
Date: Sun, 01 Aug 2021 11:48:52 GMT
Server: AmazonS3

figshare (https://ndownloader.figshare.com/files/14362754?private_link=3a8b1d84c5e197edc97c) redirects us to AWS (https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/14362754/JRC2018F_FAFB.h5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=17be44bf7ab705f3f8c19b714781f0e63277b9a10059d780a3cd7c046a489576&X-Amz-Date=20210801T113223Z&X-Amz-Credential=AKIAIYCQYOYV5JSSROOA/20210801/eu-west-1/s3/aws4_request) which is where the HEAD requests fails with a 403 Forbidden. Strangely, an equivalent GET request works just fine.

This used to work just fine so I'm assuming something changed on the AWS side. @jefferis any ideas?

@tedwu-neuro In the meantime, you can download these files manually, put them in ~/flybrain-data/ and start a fresh Python session before importing flybrains.

https://ndownloader.figshare.com/files/14362754?private_link=3a8b1d84c5e197edc97c
https://ndownloader.figshare.com/files/14368703?private_link=2a684586d5014e31076c
https://ndownloader.figshare.com/files/14369093?private_link=d5965dad295e46241ae1
https://ndownloader.figshare.com/files/21749535?private_link=ca603876efb33fdf3028
https://ndownloader.figshare.com/files/14371574?private_link=b7120207f38b35f1e372
https://ndownloader.figshare.com/files/14368358?private_link=b29e25b6e47ccf9187a8
tedwu-neuro commented 3 years ago

Hi Philipp,

Thanks for your prompt reply!

I did it manually, and it works.

However, after I tried to install the jefferislab_transforms with the code (flybrains.download_jefferislab_transforms()), I got some other error messages when I started a new Python session and then import flybrains.

--------
C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\BridgingRegistrations\Cell07_IS2.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\BridgingRegistrations\FCWB_JFRC2.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\BridgingRegistrations\IS2_Cell07.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\BridgingRegistrations\JFRC2_FCWB.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\BridgingRegistrations\JFRC2_IS2.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\BridgingRegistrations\JFRC2_JFRC2013.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\BridgingRegistrations\T1_IS2.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\Dmel_DsecI.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\Dmel_Dsim.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\Dmel_Dvir.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\Dmel_Dyak.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\Dmel_IS2.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\Dmel_JFRC2.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\DsecI_DsecF.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\DsecI_DsecM.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\DsecI_IS2.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\DsecI_JFRC2.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\Dsim_DsecI.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\IS2_DsecI.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\DrosophilidBridgingRegistrations\JFRC2_DsecI.list\original.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\MirrorRegistrations\ATAG_mirror.list\orig.list as transform: list index out of range

C:\Users\u0144374\Anaconda3\lib\site-packages\flybrains\core.py:238: UserWarning:

Error registering C:\Users\u0144374\flybrain-data\MirrorRegistrations\JFRC2_mirror.list\orig.list as transform: list index out of range
----------

and when I remove those three folders (BridgingRegistrations,DrosophilidBridgingRegistrations, MirrorRegistrations) within flybrain-date folder, I can import flybrains normally.

Do you have any idea about why?

Thanks a lot!

Best, Yifan

schlegelp commented 3 years ago

My disclaimer is that I have not tested any of the packages on Windows and you might run into issues further down the line but we can try debugging as best as possible.

Re those warnings: CMTK transforms are essentially folders like Cell07_IS2.list where Cell07 is the target and IS2 is the source. flybrains looks for folders that end with .list (for CMTK) and expects the name to be {target}_{source}.list for parsing.

As you can see from the warnings it also tries to register the orig.list or original.list. These appear to show as symlinks in Linux/OSX which is why they are skipped entirely. In Windows, they apparently to show up as file/folder which is why flybrains tries to register them as transforms and then fails because they don't follow the {target}_{source}.list convention.

When you delete the transform folders that warning goes away because there is nothing for flybrains to register.

Now to the fix:

  1. You can actually just ignore those warnings since the actual transforms should be correctly registered.
  2. I also just released a new version (0.1.10) that should fix the issue.
tedwu-neuro commented 3 years ago

I just upgrade the flybrains to the new version (0.1.10), and the issue is fixed!

Thanks a lot!

Best, Yifan