"Failed to save Label to MONAI Label Server" slicer plugin error

[rcremese]

Describe the bug When trying to save a segmentation made with the radiology app on Slicer, the following error is raised by the plugin : "Failed to save Label to MONAI Label Server" while the file is actually saved on the server. When looking at the logs on the server side, the error comes from the uvicorn package.

Specificity of my setup : MONAI Label server runs on a "Ubuntu 18.04.6 LTS" machine accessed by the client application by its http adress. The client runs on Slicer v5.0.3, on a Windows 10 machine. The data folder on which images are located is a mounted volume with cifs protocol, on the server machine. Even if an error is thrown by the server, the nifti label file is saved on the mounted volume.

To Reproduce

1. Mount a volume on a linux machine with cifs protocol
2. Download on the mounted volume the MSD Task09_Spleen dataset presented in the tutorials
3. Download the radiology app on the hard-drive of the linux machine, for ex in ~/Desktop
4. Start a MONAI Label server on the machine as presented in the quickstart tutorial

monailabel start_server -a ~/Desktop/apps/radiology -s /mnt/volume/Task09_Spleen/imagesTr/ -c models deepedit

5. Start a MONAI Label client using Slicer v5.0+ plugin and link it to the running server using the http adress of the server
6. Perform inference with the pretrained model in order to have labels
7. Click on the "Submit Label" button on the client app

Expected behavior On the client side : the progression wheel is 100% and the slicer window for label submission closed On the server side : labels are submitted to the server in a sub-directory labels/final in the image folder

Screenshots See below for screenshots of the Slicer plugin error and proof that the submitted label spleen_14.nii.gz is written on mounted volume, located on the server

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c 'import monai; monai.config.print_debug_info()'

================================ Printing MONAI config...

MONAI version: 1.0.0 Numpy version: 1.23.2 Pytorch version: 1.12.1 MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False MONAI rev id: 170093375ce29267e45681fcec09dfa856e1d7e7 MONAI file: /home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monai/init.py

Optional dependencies: Pytorch Ignite version: 0.4.10 Nibabel version: 4.0.2 scikit-image version: 0.19.3 Pillow version: 9.2.0 Tensorboard version: 2.10.0 gdown version: 4.5.1 TorchVision version: 0.13.1+cu102 tqdm version: 4.64.1 lmdb version: 1.3.0 psutil version: 5.9.2 pandas version: NOT INSTALLED or UNKNOWN VERSION. einops version: 0.4.1 transformers version: NOT INSTALLED or UNKNOWN VERSION. mlflow version: NOT INSTALLED or UNKNOWN VERSION. pynrrd version: 0.4.3

For details about installing the optional dependencies, please visit: https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================ Printing system config...

System: Linux Linux version: Ubuntu 18.04.6 LTS Platform: Linux-4.15.0-159-generic-x86_64-with-glibc2.27 Processor: x86_64 Machine: x86_64 Python version: 3.9.13 Process name: python Command: ['python', '-c', 'import monai; monai.config.print_debug_info()'] Open files: [] Num physical CPUs: 20 Num logical CPUs: 40 Num usable CPUs: 40 CPU usage (%): [8.1, 8.1, 7.5, 7.5, 8.1, 6.9, 49.1, 7.5, 7.5, 7.5, 7.5, 7.5, 8.7, 8.6, 8.1, 7.5, 7.5, 8.0, 7.5, 6.9, 6.9, 7.5, 7.6, 7.5, 9.3, 7.6, 8.0, 8.1, 9.3, 57.8, 8.7, 8.1, 8.1, 8.1, 5.6, 8.2, 11.6, 8.1, 8.1, 7.5] CPU freq. (MHz): 1336 Load avg. in last 1, 5, 15 mins (%): [1.1, 0.9, 0.9] Disk usage (%): 69.3 Avg. sensor temp. (Celsius): UNKNOWN for given OS Total physical memory (GB): 251.8 Available memory (GB): 202.4 Used memory (GB): 45.5

================================ Printing GPU config...

Num GPUs: 4 Has CUDA: True CUDA version: 11.3 cuDNN enabled: True cuDNN version: 8302 Current device: 0 Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37'] GPU 0 Name: Tesla V100-DGXS-32GB GPU 0 Is integrated: False GPU 0 Is multi GPU board: False GPU 0 Multi processor count: 80 GPU 0 Total memory (GB): 31.7 GPU 0 CUDA capability (maj.min): 7.0 GPU 1 Name: Tesla V100-DGXS-32GB GPU 1 Is integrated: False GPU 1 Is multi GPU board: False GPU 1 Multi processor count: 80 GPU 1 Total memory (GB): 31.7 GPU 1 CUDA capability (maj.min): 7.0 GPU 2 Name: Tesla V100-DGXS-32GB GPU 2 Is integrated: False GPU 2 Is multi GPU board: False GPU 2 Multi processor count: 80 GPU 2 Total memory (GB): 31.7 GPU 2 CUDA capability (maj.min): 7.0 GPU 3 Name: Tesla V100-DGXS-32GB GPU 3 Is integrated: False GPU 3 Is multi GPU board: False GPU 3 Multi processor count: 80 GPU 3 Total memory (GB): 31.7 GPU 3 CUDA capability (maj.min): 7.0

Additional context Slicer plugin error message : Traceback (most recent call last): File "C:/Users/rcremese/AppData/Local/NA-MIC/Slicer 5.0.3/NA-MIC/Extensions-30893/MONAILabel/lib/Slicer-5.0/qt-scripted-modules/MONAILabel.py", line 1408, in onSaveLabel result = self.logic.save_label(self.current_sample["id"], label_in, {"label_info": label_info}) File "C:/Users/rcremese/AppData/Local/NA-MIC/Slicer 5.0.3/NA-MIC/Extensions-30893/MONAILabel/lib/Slicer-5.0/qt-scripted-modules/MONAILabel.py", line 2141, in save_label return MONAILabelClient(self.server_url, self.tmpdir, self.client_id).save_label( File "C:\Users\rcremese\AppData\Local\NA-MIC\Slicer 5.0.3\NA-MIC\Extensions-30893\MONAILabel\lib\Slicer-5.0\qt-scripted-modules\MONAILabelLib\client.py", line 142, in save_label raise MONAILabelException( MONAILabelLib.client.MONAILabelException: (2, "Status: 500; Response: b'Internal Server Error'")

monailabel server log error message: [2022-09-20 14:38:07,522] [31780] [MainThread] [INFO] (monailabel.endpoints.datastore:100) - Saving Label for spleen_14 for tag: final by admin [2022-09-20 14:38:07,525] [31780] [MainThread] [INFO] (monailabel.endpoints.datastore:111) - Save Label params: {"label_info": [{"name": "spleen", "idx": 2}, {"name": "right kidney", "idx": 3}, {"name": "left kidney", "idx": 4}, {"name": "liver", "idx": 5}, {"name": "stomach", "idx": 6}, {"name": "aorta", "idx": 7}, {"name": "inferior vena cava", "idx": 8}], "client_id": "user-xyz"} [2022-09-20 14:38:07,525] [31780] [MainThread] [INFO] (monailabel.datastore.local:449) - Saving Label for Image: spleen_14; Tag: final; Info: {'label_info': [{'name': 'spleen', 'idx': 2}, {'name': 'right kidney', 'idx': 3}, {'name': 'left kidney', 'idx': 4}, {'name': 'liver', 'idx': 5}, {'name': 'stomach', 'idx': 6}, {'name': 'aorta', 'idx': 7}, {'name': 'inferior vena cava', 'idx': 8}], 'client_id': 'user-xyz'} [2022-09-20 14:38:07,525] [31780] [MainThread] [INFO] (monailabel.datastore.local:457) - Adding Label: spleen_14 => final => /tmp/tmpchhfnncx.nii.gz [2022-09-20 14:38:07,546] [31780] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application Traceback (most recent call last): File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi result = await app(self.scope, self.receive, self.send) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in call return await self.app(scope, receive, send) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/fastapi/applications.py", line 269, in call await super().call(scope, receive, send) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/applications.py", line 124, in call await self.middleware_stack(scope, receive, send) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in call raise exc File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in call await self.app(scope, receive, _send) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/middleware/cors.py", line 84, in call await self.app(scope, receive, send) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/exceptions.py", line 93, in call raise exc File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in call await self.app(scope, receive, sender) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call raise e File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call await self.app(scope, receive, send) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/routing.py", line 670, in call await route.handle(scope, receive, send) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/routing.py", line 266, in handle await self.app(scope, receive, send) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/starlette/routing.py", line 65, in app response = await func(request) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/fastapi/routing.py", line 227, in app raw_response = await run_endpoint_function( File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/fastapi/routing.py", line 160, in run_endpoint_function return await dependant.call(**values) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monailabel/endpoints/datastore.py", line 210, in api_save_label return save_label(background_tasks, image, params, tag, label, user.username) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monailabel/endpoints/datastore.py", line 113, in save_label label_id = instance.datastore().save_label(image, label_file, tag, save_params) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/site-packages/monailabel/datastore/local.py", line 465, in save_label shutil.copy(label_filename, dest) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/shutil.py", line 428, in copy copymode(src, dst, follow_symlinks=follow_symlinks) File "/home/rocremes/mambaforge/envs/mlabel-env/lib/python3.9/shutil.py", line 317, in copymode chmod_func(dst, stat.S_IMODE(st.st_mode)) PermissionError: [Errno 1] Operation not permitted: '/mnt/hecatonchire/PIPELINE/datas/Task09_Spleen/imagesTr/labels/final/spleen_14.nii.gz'

[tangy5]

Hi @rcremese , according to the logs, it seems MONAI label try to save label to your file system. There is an authentication issue which causing the "no response" error at Slicer.

PermissionError: [Errno 1] Operation not permitted: '/mnt/hecatonchire/PIPELINE/datas/Task09_Spleen/imagesTr/labels/final/spleen_14.nii.gz' Could you check whether you have the permission to save/write files to the system? Or chmod file system folders, I assume the sudo or root authentication is needed to write files. Thanks

[rcremese]

Hi @tangy5, I checked all my rights on folders I want to write on and user as group have read and right access to the folder. I did a little test on my side and realised I have a problem with my mounted volume configuration to copy files via python script. I should have tested it before writting this issue and I will try to figure this out by myself. Nevertheless, even if the "submit label" raised an error, the file is written to the folder /labels/final on the server due to a limitation of shutil.copy