search

Project-MONAI / MONAILabel

MONAI Label is an intelligent open source image labeling and learning tool.
https://docs.monai.io/projects/label
Apache License 2.0
610 stars 196 forks source link

[8263] [MainThread] [ERROR] (uvicorn.error:56) - Application startup failed. Exiting. #1005

Closed rcremese closed 2 years ago

rcremese commented 2 years ago

Describe the bug HTTP Error 404 : Not Found raised by the urllib when trying to download the swin_unetr_btcv_segmentation_v0.1.0 model bundle with the monaibundle app as presented in the Quickstart Tutorial 2 - Bundle App Tutorial and Use Cases.

To Reproduce Follow the Quickstart Tutorial 2 - Use Case 1 (https://docs.monai.io/projects/label/en/latest/quickstart.html#tutorial-2-bundle-app-tutorial-and-use-cases) with the attached conda environnement on a Linux system

Expected behavior Expected the server to run locally

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c 'import monai; monai.config.print_debug_info()'

================================ Printing MONAI config...

MONAI version: 0.9.1 Numpy version: 1.23.2 Pytorch version: 1.12.1 MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False MONAI rev id: 356d2d2f41b473f588899d705bbc682308cee52c MONAI file: /home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monai/init.py

Optional dependencies: Pytorch Ignite version: 0.4.9 Nibabel version: 4.0.2 scikit-image version: 0.19.3 Pillow version: 9.2.0 Tensorboard version: 2.10.0 gdown version: 4.5.1 TorchVision version: 0.13.1+cu102 tqdm version: 4.64.1 lmdb version: 1.3.0 psutil version: 5.9.2 pandas version: NOT INSTALLED or UNKNOWN VERSION. einops version: 0.4.1 transformers version: NOT INSTALLED or UNKNOWN VERSION. mlflow version: NOT INSTALLED or UNKNOWN VERSION. pynrrd version: 0.4.3

For details about installing the optional dependencies, please visit: https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================ Printing system config...

System: Linux Linux version: Ubuntu 18.04.6 LTS Platform: Linux-4.15.0-159-generic-x86_64-with-glibc2.27 Processor: x86_64 Machine: x86_64 Python version: 3.9.13 Process name: python Command: ['python', '-c', 'import monai; monai.config.print_debug_info()'] Open files: [] Num physical CPUs: 20 Num logical CPUs: 40 Num usable CPUs: 40 CPU usage (%): [11.2, 9.0, 9.0, 9.8, 40.6, 9.7, 10.4, 9.0, 9.0, 9.0, 9.8, 9.0, 9.0, 8.3, 9.0, 9.6, 9.0, 9.1, 9.7, 8.3, 9.0, 9.0, 9.0, 9.7, 11.1, 9.0, 9.7, 9.7, 9.0, 67.2, 9.8, 9.7, 9.7, 9.0, 10.7, 11.9, 9.8, 9.7, 9.0, 9.6] CPU freq. (MHz): 1271 Load avg. in last 1, 5, 15 mins (%): [0.9, 0.9, 0.9] Disk usage (%): 66.3 Avg. sensor temp. (Celsius): UNKNOWN for given OS Total physical memory (GB): 251.8 Available memory (GB): 202.4 Used memory (GB): 45.5

================================ Printing GPU config...

Num GPUs: 4 Has CUDA: True CUDA version: 11.3 cuDNN enabled: True cuDNN version: 8302 Current device: 0 Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37'] GPU 0 Name: Tesla V100-DGXS-32GB GPU 0 Is integrated: False GPU 0 Is multi GPU board: False GPU 0 Multi processor count: 80 GPU 0 Total memory (GB): 31.7 GPU 0 CUDA capability (maj.min): 7.0 GPU 1 Name: Tesla V100-DGXS-32GB GPU 1 Is integrated: False GPU 1 Is multi GPU board: False GPU 1 Multi processor count: 80 GPU 1 Total memory (GB): 31.7 GPU 1 CUDA capability (maj.min): 7.0 GPU 2 Name: Tesla V100-DGXS-32GB GPU 2 Is integrated: False GPU 2 Is multi GPU board: False GPU 2 Multi processor count: 80 GPU 2 Total memory (GB): 31.7 GPU 2 CUDA capability (maj.min): 7.0 GPU 3 Name: Tesla V100-DGXS-32GB GPU 3 Is integrated: False GPU 3 Is multi GPU board: False GPU 3 Multi processor count: 80 GPU 3 Total memory (GB): 31.7 GPU 3 CUDA capability (maj.min): 7.0

Additional context The following is the output of the command monailabel start_server -a -s <path to Task09_Spleen/imagesTr> --conf models swin_unetr_btcv_segmentation_v0.1.0 : Using PYTHONPATH=/home/rocremes/mambaforge/envs: [2022-09-19 14:01:55,861] [11037] [MainThread] [INFO] (main:269) - USING:: app = /home/rocremes/projects/monai-label/apps/monaibundle [2022-09-19 14:01:55,861] [11037] [MainThread] [INFO] (main:269) - USING:: studies = /mnt/hecatonchire/PIPELINE/datas/Task09_Spleen/imagesTr [2022-09-19 14:01:55,861] [11037] [MainThread] [INFO] (main:269) - USING:: verbose = INFO [2022-09-19 14:01:55,861] [11037] [MainThread] [INFO] (main:269) - USING:: conf = [['models', 'swin_unetr_btcv_segmentation_v0.1.0']] [2022-09-19 14:01:55,861] [11037] [MainThread] [INFO] (main:269) - USING:: host = 0.0.0.0 [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: port = 8000 [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: uvicorn_app = monailabel.app:app [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: ssl_keyfile = None [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: ssl_certfile = None [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: ssl_keyfile_password = None [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: ssl_ca_certs = None [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: workers = None [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: limit_concurrency = None [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: access_log = False [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: log_config = None [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: dryrun = False [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:269) - USING:: action = start_server [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_API_STR = [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_APP_DIR = [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_STUDIES = [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_AUTH_DB = [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}' [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE = [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_USERNAME = [2022-09-19 14:01:55,862] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PASSWORD = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_API_KEY = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_CACHE_PATH = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PROJECT = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_ASSET_PATH = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_DSA_ANNOTATION_GROUPS = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '[".nii.gz", ".nii", ".nrrd", ".jpg", ".png", ".tif", ".svs", ".xml"]' [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000 [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]' [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH = [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600 [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1 [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600 [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True [2022-09-19 14:01:55,863] [11037] [MainThread] [INFO] (main:274) - Allow Origins: ['*'] [2022-09-19 14:01:56,310] [11037] [MainThread] [INFO] (uvicorn.error:75) - Started server process [11037] [2022-09-19 14:01:56,310] [11037] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup. [2022-09-19 14:01:56,310] [11037] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: /home/rocremes/projects/monai-label/apps/monaibundle; studies: /mnt/hecatonchire/PIPELINE/datas/Task09_Spleen/imagesTr; conf: {'models': 'swin_unetr_btcv_segmentation_v0.1.0'} [2022-09-19 14:01:56,312] [11037] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for MONAILabelApp Found: <class 'main.MyApp'> [2022-09-19 14:01:56,542] [11037] [MainThread] [INFO] (main:78) - +++ Adding Model: swin_unetr_btcv_segmentation_v0.1.0 => {'checksum': '65eaf535f04e80c8d82c597449a60ef8c1622692', 'source': 'https://github.com/Project-MONAI/model-zoo/releases/download/hosting_storage_v1/swin_unetr_btcv_segmentation_v0.1.0.zip'} => /home/rocremes/projects/monai-label/apps/monaibundle/model/swin_unetr_btcv_segmentation_v0.1.0 2022-09-19 14:01:56,543 - INFO - --- input summary of monai.bundle.scripts.download --- 2022-09-19 14:01:56,543 - INFO - > name: 'swin_unetr_btcv_segmentation_v0.1.0' 2022-09-19 14:01:56,543 - INFO - > bundle_dir: '/home/rocremes/projects/monai-label/apps/monaibundle/model' 2022-09-19 14:01:56,543 - INFO - > source: 'github' 2022-09-19 14:01:56,543 - INFO - > repo: 'Project-MONAI/model-zoo/ll' 2022-09-19 14:01:56,543 - INFO - > progress: True 2022-09-19 14:01:56,543 - INFO - ---

swin_unetr_btcv_segmentation_v0.1.0.zip: 0.00B [00:00, ?B/s] 2022-09-19 14:01:56,790 - ERROR - Download failed from https://github.com/Project-MONAI/model-zoo/releases/download/ll/swin_unetr_btcv_segmentation_v0.1.0.zip to /tmp/tmp7ik_1nmn/swin_unetr_btcv_segmentation_v0.1.0.zip. [2022-09-19 14:01:56,793] [11037] [MainThread] [ERROR] (uvicorn.error:119) - Traceback (most recent call last): File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/routing.py", line 635, in lifespan async with self.lifespan_context(app): File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/routing.py", line 530, in aenter await self._router.startup() File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/routing.py", line 612, in startup await handler() File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monailabel/app.py", line 104, in startup_event instance = app_instance() File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 51, in app_instance app = c(app_dir=app_dir, studies=studies, conf=conf) File "/home/rocremes/projects/monai-label/apps/monaibundle/main.py", line 80, in init download(name=k, bundle_dir=self.model_dir, source=zoo_source, repo=zoo_repo) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monai/bundle/scripts.py", line 223, in download _download_fromgithub(repo=repo, download_path=bundledir, filename=name, progress=progress) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monai/bundle/scripts.py", line 140, in _download_from_github download_url(url=url, filepath=filepath, hash_val=None, progress=progress) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monai/apps/utils.py", line 203, in download_url _download_with_progress(url, tmp_name, progress=progress) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monai/apps/utils.py", line 114, in _download_with_progress raise e File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monai/apps/utils.py", line 107, in _download_with_progress urlretrieve(url, filepath, reporthook=t.update_to) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/urllib/request.py", line 239, in urlretrieve with contextlib.closing(urlopen(url, data)) as fp: File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/urllib/request.py", line 214, in urlopen return opener.open(url, data, timeout) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/urllib/request.py", line 523, in open response = meth(req, response) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/urllib/request.py", line 632, in http_response response = self.parent.error( File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/urllib/request.py", line 561, in error return self._call_chain(args) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/urllib/request.py", line 494, in _call_chain result = func(args) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/urllib/request.py", line 641, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 404: Not Found

[2022-09-19 14:01:56,793] [11037] [MainThread] [ERROR] (uvicorn.error:56) - Application startup failed. Exiting.

mlabel_env.txt

diazandr3s commented 2 years ago

Hi @rcremese,

Thanks for opening this issue. Can I ask you which MONAI Label version are you using? Indeed the link to the pretrained model has changed to this one: https://github.com/Project-MONAI/model-zoo/releases/download/hosting_storage_v1/swin_unetr_btcv_segmentation_v0.1.0.zip and it is updated on the latest MONAI Label version: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/monaibundle/main.py#L36

@yuchen-xu can you please confirm this?

tangy5 commented 2 years ago

@rcremese , the MONAI Label need to first download the Model-Zoo bundle model first, I saw the error log, the automatic download seems failed. As @diazandr3s mentioned, could you try download from the link above. Save , unzip the model to And start MONAI Laberl server with the downloade bundle. Thanks for using Bundle App in the MONAI label.

rcremese commented 2 years ago

Hello ! Thanks for your kick reply. My current version of monailabel is v0.4.2 so I forgot to update it before trying the monaibundle. @tangy5 I tried to download and unzip the swin_unetr_btcv_segmentation_v0.1.0.zip bundle into the model folder of the monaibundle app but this didn't overide the download procedure

tangy5 commented 2 years ago

Let me dig into this, and get reply to you once I figure out, thank you.

rcremese commented 2 years ago

Ok šŸ‘ I updated my monailabel version to v0.5 but still have the same problem where the script fetches the model at the wrong adress : 2022-09-19 17:10:00,965 - ERROR - Download failed from https://github.com/Project-MONAI/model-zoo/releases/download/ll/swin_unetr_btcv_segmentation_v0.1.0.zip to /tmp/tmpok6ca95y/swin_unetr_btcv_segmentation_v0.1.0.zip

I can join my current conda environnement if needed

tangy5 commented 2 years ago

Hi @rcremese , thanks for verifying the version, unless MONAI >=0.9.1 and MONAI Label >=0.4.2, it will support Bundle app and integration. I just tired the run on my linux machine, the command should be fine. I check and compared my logs with your posted logs, I found the MONAI_ZOO_REPO variable is different. On your logs: it's

2022-09-19 14:01:56,543 - INFO - > repo: 'Project-MONAI/model-zoo/ll'

The default MONAI_ZOO_REPO should be:

MONAI_ZOO_REPO = "Project-MONAI/model-zoo/hosting_storage_v1"

This is a path for referring to the correct remote model zoo service. Not for a local workspace.

Can you try put MONAI_ZOO_REPO = "Project-MONAI/model-zoo/hosting_storage_v1" in the main.py ?

https://github.com/Project-MONAI/MONAILabel/blob/5a7cbd8d5de945a99085d0f6495de21485c47415/sample-apps/monaibundle/main.py#L36

For double check: these lines in main.py at monaibundle app should be set default as follows:

MONAI_ZOO_INFO = "https://raw.githubusercontent.com/Project-MONAI/model-zoo/dev/models/model_info.json"
MONAI_ZOO_SOURCE = "github"
MONAI_ZOO_REPO = "Project-MONAI/model-zoo/hosting_storage_v1"

let me know if this work or not. Thanks.

rcremese commented 2 years ago

Ok, I see, I just realised how dumb I am šŸ˜“ I probly changed this variable when looking at the main.py file and didn't even realised ! Sorry for bothering you. Everything works fine now !

tangy5 commented 2 years ago

Ok, I see, I just realised how dumb I am sweat I probly changed this variable when looking at the main.py file and didn't even realised ! Sorry for bothering you. Everything works fine now !

No problem, thank you for using MONAI label and the bundle app, the Bundle App is a new feature added recently. Let us know if you have any questions or feedback. It's an open project, and we'd appreciated all suggestions and user experience comment!

rcremese commented 2 years ago

I will try to give you feedback ! I'm super excited to try it out. By the way, thank you for the great work