search

Deep-MI / FastSurfer

PyTorch implementation of FastSurferCNN
Apache License 2.0
435 stars 115 forks source link

Model download issue #518

Closed roohy466 closed 4 weeks ago

roohy466 commented 1 month ago

Hi, Seems the latest version of run_fastsurfer has issue downloading the models:

[INFO: run_prediction.py:  652]: Checking or downloading default checkpoints ...
[INFO: checkpoint.py:  341]: Downloading checkpoint aparc_vinn_axial_v2.0.0.pkl from https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59
[WARNING: checkpoint.py:  351]: Server https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59 not reachable.
[INFO: checkpoint.py:  341]: Downloading checkpoint aparc_vinn_axial_v2.0.0.pkl from https://zenodo.org/records/10390573/files
[WARNING: checkpoint.py:  351]: Server https://zenodo.org/records/10390573/files not reachable.
Traceback (most recent call last):
  File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/run_prediction.py", line 806, in <module>
    sys.exit(main(**vars(_args)))
  File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/run_prediction.py", line 656, in main
    get_checkpoints(ckpt_ax, ckpt_cor, ckpt_sag, urls=urls)
  File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/checkpoint.py", line 397, in get_checkpoints
    check_and_download_ckpts(file, urls)
  File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/checkpoint.py", line 379, in check_and_download_ckpts
    download_checkpoint(checkpoint_path.name, checkpoint_path, urls)
  File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/checkpoint.py", line 354, in download_checkpoint
    raise requests.exceptions.RequestException("No server reachable.")
requests.exceptions.RequestException: No server reachable.
ERROR: FastSurfer asegdkt segmentation failed.
dkuegler commented 1 month ago

Thank you for reporting this issue!

Edit: sorry, I misread the message. Can you please verify that

  1. The problem still persists
  2. you are able to reach zenodo.org from where you called Fastsurfer?

We specifically implemented code in #467 and #488 so that we have two servers from which to download the model weights.

dkuegler commented 1 month ago

@agirodi can you please assist, if the problem persists?

roohy466 commented 1 month ago

The problem still persists but I worked around by manually downloading the models from https://zenodo.org and put them in the checkpoints folder. I was able to wget the files directly through zenodo but Fastsurfer is giving above errors!

dkuegler commented 1 month ago

Thank you again for reporting this issue, I don't really know how we can reproduce this issue though.

Could you check for me, if download via the download_checkpoints.py script works for you? Basically just run PYTHONPATH=. python FastSurferCNN/download_checkpoints.py --all ... That is what I am using on a regular basis to download checkpoints when I am building docker Images for tests. And I have not recently experienced any errors.

@agirodi can you maybe check, if there is an issue with downloading checkpoints as part of the run_prediction.py script?

roohy466 commented 1 month ago

I am getting another error:

(fastsurfer) ubuntu@32d172313004:~/data$ /home/ubuntu/python-envs/fastsurfer/bin/python3 /usr/local/KUL_apps/FastSurfer/FastSurferCNN/download_checkpoints.py --all
Traceback (most recent call last):
  File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/download_checkpoints.py", line 21, in <module>
    from FastSurferCNN.utils import PLANES
ModuleNotFoundError: No module named 'FastSurferCNN'
dkuegler commented 1 month ago

That error means the PYTHONPATH was not configured correctly. I put PYTHONPATH=. python FastSurferCNN/download_checkpoints.py --all assuming you would be in the fastsurfer root directory so you should do PYTHONPATH=/usr/local/KUL_apps/FastSurfer python FastSurferCNN/download_checkpoints.py --all

agirodi commented 1 month ago

I wasn't able to replicate this error either. The checkpoints download as intended for me. According to the error message, both servers are unreachable.The additional logging from #519 may help debug why your pc can not reach the servers. Maybe try curl https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59 to check connection to the server

m-reuter commented 1 month ago

Probably something restricting internet access from that computer or docker container.

dkuegler commented 1 month ago

Could also be something like outdated root certificates #519 would help debug like @agirodi said.

roohy466 commented 1 month ago

That error means the PYTHONPATH was not configured correctly. I put PYTHONPATH=. python FastSurferCNN/download_checkpoints.py --all assuming you would be in the fastsurfer root directory so you should do PYTHONPATH=/usr/local/KUL_apps/FastSurfer python FastSurferCNN/download_checkpoints.py --all

After installing some packages like torch, requests, yacs, nibabel again! I got this error:

  (fastsurfer) ubuntu@ab818f2ef143:/usr/local/KUL_apps/FastSurfer$ PYTHONPATH=/usr/local/KUL_apps/FastSurfer python FastSurferCNN/download_checkpoints.py --all
  INFO:FastSurferCNN.utils.checkpoint:Downloading checkpoint aparc_vinn_axial_v2.0.0.pkl from https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59
  WARNING:FastSurferCNN.utils.checkpoint:Server https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59 not reachable.
  INFO:FastSurferCNN.utils.checkpoint:Downloading checkpoint aparc_vinn_axial_v2.0.0.pkl from https://zenodo.org/records/10390573/files
  WARNING:FastSurferCNN.utils.checkpoint:Server https://zenodo.org/records/10390573/files not reachable.
  Traceback (most recent call last):
    File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/download_checkpoints.py", line 106, in main
      get_checkpoints(
    File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/checkpoint.py", line 397, in get_checkpoints
      check_and_download_ckpts(file, urls)
    File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/checkpoint.py", line 379, in check_and_download_ckpts
      download_checkpoint(checkpoint_path.name, checkpoint_path, urls)
    File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/checkpoint.py", line 354, in download_checkpoint
      raise requests.exceptions.RequestException("No server reachable.")
  requests.exceptions.RequestException: No server reachable.
  No server reachable.
roohy466 commented 1 month ago

curl https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59

I get this error:

(fastsurfer) ubuntu@ab818f2ef143:/usr/local/KUL_apps/FastSurfer$ curl https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59
{
  "contents": [
    {
      "version_id": "fe1a8734-cfd7-4cd3-b842-556d8de7ccce", 
      "links": {
        "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_axial_v1.0.0.pkl", 
        "version": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_axial_v1.0.0.pkl?versionId=fe1a8734-cfd7-4cd3-b842-556d8de7ccce", 
        "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_axial_v1.0.0.pkl?uploads"
      }, 
      "checksum": "md5:3272d2ed82f542f0c38323b160974f01", 
      "size": 21677617, 
      "created": "2022-09-28T17:20:35.505984+00:00", 
      "key": "aparc_cnn_axial_v1.0.0.pkl", 
      "mimetype": "application/octet-stream", 
      "is_head": true, 
      "delete_marker": false, 
      "updated": "2022-09-28T17:20:35.509987+00:00"
    }, 
    {
      "version_id": "bcd99798-6973-42f4-ae03-513c9d7a1a80", 
      "links": {
        "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_axial_v2.0.0.pkl", 
        "version": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_axial_v2.0.0.pkl?versionId=bcd99798-6973-42f4-ae03-513c9d7a1a80", 
        "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_axial_v2.0.0.pkl?uploads"
      }, 
      "checksum": "md5:55df5bca0324e939664e0133a2e26e0c", 
      "size": 19387352, 
      "created": "2022-09-28T17:20:35.514541+00:00", 
      "key": "aparc_cnn_axial_v2.0.0.pkl", 
      "mimetype": "application/octet-stream", 
      "is_head": true, 
      "delete_marker": false, 
      "updated": "2022-09-28T17:20:35.518067+00:00"
    }, 
    {
      "version_id": "afbe367b-7b63-4567-8918-659f1349d3a4", 
      "links": {
        "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_coronal_v1.0.0.pkl", 
        "version": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_coronal_v1.0.0.pkl?versionId=afbe367b-7b63-4567-8918-659f1349d3a4", 
        "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_coronal_v1.0.0.pkl?uploads"
      }, 
      "checksum": "md5:8b7e1b1991b1859e5141723ffcb04f93", 
      "size": 21677617, 
      "created": "2022-09-28T17:20:35.522530+00:00", 
      "key": "aparc_cnn_coronal_v1.0.0.pkl", 
      "mimetype": "application/octet-stream", 
      "is_head": true, 
      "delete_marker": false, 
      "updated": "2022-09-28T17:20:35.525964+00:00"
    }, 
    {
      "version_id": "1a219c1c-4c36-4bf3-8e28-3d8ff24acdd2", 
      "links": {
        "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_coronal_v2.0.0.pkl", 
        "version": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_coronal_v2.0.0.pkl?versionId=1a219c1c-4c36-4bf3-8e28-3d8ff24acdd2", 
        "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_coronal_v2.0.0.pkl?uploads"
      }, 
      "checksum": "md5:5c684410eb0b33e4f1c680ee08271a88", 
      "size": 19386908, 
      "created": "2022-09-28T17:20:35.530326+00:00", 
      "key": "aparc_cnn_coronal_v2.0.0.pkl", 
      "mimetype": "application/octet-stream", 
      "is_head": true, 
      "delete_marker": false, 
      "updated": "2022-09-28T17:20:35.533967+00:00"
    }, 
    {
      "version_id": "d80fdf13-0363-4d7b-897c-32a768c7bbd5", 
      "links": {
        "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_sagittal_v1.0.0.pkl", 
        "version": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_sagittal_v1.0.0.pkl?versionId=d80fdf13-0363-4d7b-897c-32a768c7bbd5", 
        "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_sagittal_v1.0.0.pkl?uploads"
      }, 
      "checksum": "md5:36f1475107c0114b5f5162890721485c", 
      "size": 21655777, 
      "created": "2022-09-28T17:20:35.538296+00:00", 
      "key": "aparc_cnn_sagittal_v1.0.0.pkl", 
      "mimetype": "application/octet-stream", 
      "is_head": true, 
      "delete_marker": false, 
      "updated": "2022-09-28T17:20:35.541770+00:00"
    }, 
    {
      "version_id": "387f8fe7-aa8a-4a7c-8975-b941edefb2f5", 
      "links": {
        "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_sagittal_v2.0.0.pkl", 
        "version": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_sagittal_v2.0.0.pkl?versionId=387f8fe7-aa8a-4a7c-8975-b941edefb2f5", 
        "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_cnn_sagittal_v2.0.0.pkl?uploads"
      }, 
      "checksum": "md5:d9eb9895c72df4d901c5af934e885f79", 
      "size": 19363182, 
      "created": "2022-09-28T17:20:35.546079+00:00", 
      "key": "aparc_cnn_sagittal_v2.0.0.pkl", 
      "mimetype": "application/octet-stream", 
      "is_head": true, 
      "delete_marker": false, 
      "updated": "2022-09-28T17:20:35.549534+00:00"
    }, 
    {
      "version_id": "b62b4e5c-a190-47ed-b443-049694a4dcbc", 
      "links": {
        "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_vinn_axial_v2.0.0.pkl", 
        "version": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_vinn_axial_v2.0.0.pkl?versionId=b62b4e5c-a190-47ed-b443-049694a4dcbc", 
        "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_vinn_axial_v2.0.0.pkl?uploads"
      }, 
      "checksum": "md5:f16e78dd069eed2ec659674e0fdb697b", 
      "size": 22399769, 
      "created": "2022-09-28T17:20:35.553951+00:00", 
      "key": "aparc_vinn_axial_v2.0.0.pkl", 
      "mimetype": "application/octet-stream", 
      "is_head": true, 
      "delete_marker": false, 
      "updated": "2022-09-28T17:20:35.557494+00:00"
    }, 
    {
      "version_id": "345c14ed-7341-41fe-a9dd-560b0f8c064f", 
      "links": {
        "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_vinn_coronal_v2.0.0.pkl", 
        "version": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_vinn_coronal_v2.0.0.pkl?versionId=345c14ed-7341-41fe-a9dd-560b0f8c064f", 
        "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_vinn_coronal_v2.0.0.pkl?uploads"
      }, 
      "checksum": "md5:fa699af3d60ca9ee06950e62ed04c288", 
      "size": 22399783, 
      "created": "2022-09-28T17:20:35.562209+00:00", 
      "key": "aparc_vinn_coronal_v2.0.0.pkl", 
      "mimetype": "application/octet-stream", 
      "is_head": true, 
      "delete_marker": false, 
      "updated": "2022-09-28T17:20:35.565705+00:00"
    }, 
    {
      "version_id": "d8243717-7b63-4491-825c-8e193d9504e9", 
      "links": {
        "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_vinn_sagittal_v2.0.0.pkl", 
        "version": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_vinn_sagittal_v2.0.0.pkl?versionId=d8243717-7b63-4491-825c-8e193d9504e9", 
        "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59/aparc_vinn_sagittal_v2.0.0.pkl?uploads"
      }, 
      "checksum": "md5:f28ab87df75492ad91f9ff5e49a2583c", 
      "size": 22375598, 
      "created": "2022-09-28T17:20:35.569951+00:00", 
      "key": "aparc_vinn_sagittal_v2.0.0.pkl", 
      "mimetype": "application/octet-stream", 
      "is_head": true, 
      "delete_marker": false, 
      "updated": "2022-09-28T17:20:35.573336+00:00"
    }
  ], 
  "links": {
    "self": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59", 
    "versions": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59?versions", 
    "uploads": "https://b2share.fz-juelich.de/api/files/a423a576-220d-47b0-9e0c-b5b32d45fc59?uploads"
  }, 
  "locked": true, 
  "size": 190323603, 
  "created": "2022-09-28T17:20:35.498962+00:00", 
  "max_file_size": 53687091200, 
  "id": "a423a576-220d-47b0-9e0c-b5b32d45fc59", 
  "quota_size": 214748364800, 
  "updated": "2022-09-28T17:20:35.575229+00:00"
roohy466 commented 1 month ago

Probably something restricting internet access from that computer or docker container.

That could be as in our company we use "zscaler" but I have added the needed certificate to the docker!

m-reuter commented 1 month ago

The curl result is not an error, but shows that it works.

Try this next (each command in a separate line). Go do a new empty directory. First checkout our dev version:

git clone https://github.com/Deep-MI/FastSurfer.git

Then go into the FastSurfer directory and set the python path:

cd FastSurfer
export PYTHONPATH=$PWD

Assuming you have python3.10 and dependencies like requests, torch, yacs, nibabel installed, then try downloading the checkpoints with:

python3.10 FastSurferCNN/download_checkpoints.py --all

This should then download them (just tested on my MacBook).

dkuegler commented 1 month ago

Hi can you also check, if your user can create folders in the fastsurfer root directory (e.g. "checkpoints") and if the user can write to the checkpoints directory? Maybe the write of the file fails?

roohy466 commented 1 month ago

Yes I did, and everything still has problem. I gave up and downloaded them manually but then when I run "run_fastsurfer.py", I got an error related to matplotlib.cm . seem cm does not have anymore the get_cmap function. 

    File "/usr/local/KUL_apps/FastSurfer/CerebNet/run_prediction.py", line 28, in <module>
      from CerebNet.inference import Inference
    File "/usr/local/KUL_apps/FastSurfer/CerebNet/inference.py", line 29, in <module>
      from FastSurferCNN.utils.mapper import JsonColorLookupTable, TSVLookupTable
    File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/mapper.py", line 52, in <module>
      from matplotlib.cm import get_cmap
  ImportError: cannot import name 'get_cmap' from 'matplotlib.cm' (/opt/miniconda/lib/python3.10/site-packages/matplotlib/cm.py)
  ERROR: Cerebellum Segmentation failed

and I replaced it with pyplot and it worked: 

  sed -i 's/from matplotlib.cm/from matplotlib.pyplot /g' /usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/mapper.py
dkuegler commented 1 month ago

Thank you for reporting this issue. I think this is a change due to incompatible matplotlib versions (1 vs. 2).

I will take a look and see if this needs changing as well.

I am sorry this issue has not been easily resolved. As stated above , we still cannot reproduce your problem and also have no clear indication on what actually went wrong in your case...

dkuegler commented 1 month ago

Yes I did, and everything still has problem. I gave up and downloaded them manually but then when I run "run_fastsurfer.py", I got an error related to matplotlib.cm . seem cm does not have anymore the get_cmap function.

    File "/usr/local/KUL_apps/FastSurfer/CerebNet/run_prediction.py", line 28, in <module>
      from CerebNet.inference import Inference
    File "/usr/local/KUL_apps/FastSurfer/CerebNet/inference.py", line 29, in <module>
      from FastSurferCNN.utils.mapper import JsonColorLookupTable, TSVLookupTable
    File "/usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/mapper.py", line 52, in <module>
      from matplotlib.cm import get_cmap
  ImportError: cannot import name 'get_cmap' from 'matplotlib.cm' (/opt/miniconda/lib/python3.10/site-packages/matplotlib/cm.py)
  ERROR: Cerebellum Segmentation failed

and I replaced it with pyplot and it worked:

  sed -i 's/from matplotlib.cm/from matplotlib.pyplot /g' /usr/local/KUL_apps/FastSurfer/FastSurferCNN/utils/mapper.py

This particular issue is based on the fact that matplotlib 3.9 has removed matplotlib.cm.get_cmap. should be fixed in #522 .

m-reuter commented 4 weeks ago

Hi, generally it is important that you use the versions of the dependencies that we also use. Which is why we provide the docker images and singularity. Native install is always tricky as we do not know what is on your system and this can break things.

Since the matplotlib incompatibility is fixed now and the download problem is caused by your setup not our servers, I will close this for now. Happy to re-open if there is more information for debugging this. Or start another issue if you run into anything else.

constructor-s commented 2 weeks ago

On a native M1 installation, I had issue downloading from b2share.fz-juelich.de using the stable branch (818acf1) and had to switch to the dev branch (3622c13).

m-reuter commented 2 weeks ago

I currently also have issues downloading from juelich even using the dev branch. Probably theirs servers are down and you just got lucky in between.

phwegner commented 4 days ago

I had the same problem downloading from the jülich server on the stable branch. 

requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='b2share.fz-juelich.de', port=443): Max retries exceeded
dkuegler commented 4 days ago

Unfortunately, the hosting service by Forschungszentrum Jülich is currently down. Unfortunately, this is a very reoccuring situation and we have implemented a fallback to zenodo in dev. So you will have to wait for the server to come back up or switch to dev.