search

Project-MONAI / monai-deploy

MONAI Deploy aims to become the de-facto standard for developing, packaging, testing, deploying and running medical AI applications in clinical production.
Apache License 2.0
98 stars 22 forks source link

MAP built with App SDK v0.6 fails when running on MONAI Deploy Express #147

Closed MMelQin closed 10 months ago

MMelQin commented 10 months ago

Observed: Configured a workflow to run an example map built with MONAI Deploy App SDK v0.6, e.g., ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0, but workflow instance failed with "no output"

Expected: MD Express should support the MAPs built with App SDK v0.6 and after; these MAPs are built differently than those with MONAI Deploy App SDK v0.5.x, and have additional requirements on the permission on the host folders (which are to be mapped as MAP output"

Preliminary Analysis: Root cause is in the writer permission, or the lack thereof, on the output folder in the host file system.

Workaround attempted but did not working due to TM/Docker behavior:

MD Express TaskManager container itself maps host volume to its internal folder, and also uses them for MAP input and output, so tried to first create the host folder with the logged-on user's security context, e.g. mkdir -p .md/mdtm. However, when the TaskManager creates further sub-folder for a triggered task, the folder is created as owned by root, without sufficient permission for the MAP to write output to. 

(.test) mqin@mingq-dt:~/md-express/deploy/monai-deploy-express$ ll .md
total 32
drwxrwxr-x   8 mqin             mqin 4096 Oct 24 19:17 ./
drwxr-xr-x   8 mqin             mqin 4096 Oct 24 19:16 ../
drwxr-xr-x   4 root             root 4096 Oct 24 19:17 mdig/
drwxrwxr-x   2 mqin             mqin 4096 Oct 24 19:16 mdtm/
drwxr-xr-x   4 root             root 4096 Oct 24 19:16 minio/
drwxr-xr-x   4 systemd-coredump root 4096 Oct 24 19:17 mongodb/
drwxr-xr-x 231 root             root 4096 Oct 24 19:18 orthanc/
drwxr-xr-x   4 systemd-coredump root 4096 Oct 24 19:17 rabbitmq/
(.test) mqin@mingq-dt:~/md-express/deploy/monai-deploy-express$ ll .md/mdtm
total 12
drwxrwxr-x 3 mqin mqin 4096 Oct 24 19:19 ./
drwxrwxr-x 8 mqin mqin 4096 Oct 24 19:17 ../
drwxr-xr-x 4 root root 4096 Oct 24 19:19 960636a5-d7be-415c-9fe3-1d93b0e783c6/

Step to reproduce: Create an workflow definition as below (note, the attributes starting with env_ do not matter as the actual values are all correct and the MAP has default env vars. However, different error happened, see Additional Info) Download the MD Express, docker compose up Register the workflow, per instruction on MD Express Readme Upload the a CT abdomen series, per MD Express Readme Send the series to DICOM device destination, monai-deploy Examine the console output of docker compose

{
        "name": "ai-liver-seg-2",
        "version": "1.0.0",
        "description": "AI Liver Segmentation - 2",
        "informatics_gateway": {
                "ae_title": "MONAI-DEPLOY",
                "data_origins": [
                        "ORTHANC"
                ],
                "export_destinations": [
                        "ORTHANC"
                ]
        },
        "tasks": [
                {
                        "id": "router",
                        "description": "Ensure series description contains liver",
                        "type": "router",
                        "task_destinations": [
                                {
                                        "name": "liver",
                                        "conditions": ["{{ context.dicom.series.any('0008','103E')}} == 'CT series for liver tumor from nii 014'"]
                                }
                        ]
                },
                {
                        "id": "liver",
                        "description": "Execute Liver Segmentation MAP",
                        "type": "docker",
                        "args": {
                                "container_image": "ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0",
                                "server_url": "unix:///var/run/docker.sock",
                                "entrypoint": "/bin/bash,-c",
                                "command": "python3 -u /opt/holoscan/app/app.py",
                                "task_timeout_minutes": "5",
                                "temp_storage_container_path": "/var/lib/monai/",
                                "env_MONAI_INPUTPATH": "/var/holoscan/input/",
                                "env_MONAI_OUTPUTPATH": "/var/holoscan/output/",
                                "env_MONAI_MODELPATH": "/opt/holoscan/models/",
                                "env_MONAI_WORKDIR": "/var/holoscan/"
                        },
                        "artifacts": {
                                "input": [
                                        {
                                                "name": "env_MONAI_INPUTPATH",
                                                "value": "{{ context.input.dicom }}"
                                        }
                                ],
                                "output": [
                                        {
                                                "name": "env_MONAI_OUTPUTPATH",
                                                "mandatory": true
                                        }
                                ]
                        },
                        "task_destinations": [
                                {
                                        "name": "export-liver-seg"
                                }
                        ]
                },
                {
                        "id": "export-liver-seg",
                        "description": "Export Segmentation Storage Object",
                        "type": "export",
                        "export_destinations": [
                                {
                                        "Name": "ORTHANC"
                                }
                        ],
                        "artifacts": {
                                "input": [
                                        {
                                                "name": "export-dicom",
                                                "value": "{{ context.executions.liver.artifacts.env_MONAI_OUTPUTPATH }}",
                                                "mandatory": true
                                        }
                                ],
                                "output": []
                        }
                }
        ]
}

Additional Info: Used a config that replaces all monai with holoscan, but encountered a different error. This one is because that in the docker-compose.yml file, the TaskManager container itself has the volume mapping set to its own /var/lib/monai/, so the workflow def cannot use "temp_storage_container_path": "/var/lib/holoscan/"

So, stick with "temp_storage_container_path": "/var/lib/monai/", and change it in the following config, to avoid this other binding error.

{
        "name": "ai-liver-seg-HS",
        "version": "1.0.0",
        "description": "AI Liver Segmentation - HS",
        "informatics_gateway": {
                "ae_title": "MONAI-DEPLOY",
                "data_origins": [
                        "ORTHANC"
                ],
                "export_destinations": [
                        "ORTHANC"
                ]
        },
        "tasks": [
                {
                        "id": "router",
                        "description": "Ensure series description contains liver",
                        "type": "router",
                        "task_destinations": [
                                {
                                        "name": "liver",
                                        "conditions": ["{{ context.dicom.series.any('0008','103E')}} == 'CT series for liver tumor from nii 014'"]
                                }
                        ]
                },
                {
                        "id": "liver",
                        "description": "Execute Liver Segmentation MAP",
                        "type": "docker",
                        "args": {
                                "container_image": "ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0",
                                "server_url": "unix:///var/run/docker.sock",
                                "entrypoint": "/bin/bash,-c",
                                "command": "python3 -u /opt/holoscan/app/app.py",
                                "task_timeout_minutes": "5",
                                "temp_storage_container_path": "/var/lib/holoscan/",
                                "env_HOLOSCAN_INPUT_PATH": "/var/holoscan/input/",
                                "env_HOLOSCAN_OUTPUT_PATH": "/var/holoscan/output/",
                                "env_HOLOSCAN_MODEL_PATH": "/opt/holoscan/models/",
                                "env_HOLOSCAN_WORKDIR": "/var/holoscan/"
                        },
                        "artifacts": {
                                "input": [
                                        {
                                                "name": "env_HOLOSCAN_INPUT_PATH",
                                                "value": "{{ context.input.dicom }}"
                                        }
                                ],
                                "output": [
                                        {
                                                "name": "env_HOLOSCAN_OUTPUT_PATH",
                                                "mandatory": true
                                        }
                                ]
                        },
                        "task_destinations": [
                                {
                                        "name": "export-liver-seg"
                                }
                        ]
                },
                {
                        "id": "export-liver-seg",
                        "description": "Export Segmentation Storage Object",
                        "type": "export",
                        "export_destinations": [
                                {
                                        "Name": "ORTHANC"
                                }
                        ],
                        "artifacts": {
                                "input": [
                                        {
                                                "name": "export-dicom",
                                                "value": "{{ context.executions.liver.artifacts.env_HOLOSCAN_OUTPUT_PATH }}",
                                                "mandatory": true
                                        }
                                ],
                                "output": []
                        }
                }
        ]
}

But got a different error from task manager on binding the volumes

mdl-tm        | 2023-10-25 00: 52: 09.7745|1010|DEBUG|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|source=1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm, target=/var/lib/holoscan/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH/1.2.826.0.1.3680043.2.1125.1.19616861412188316212577695277886020/1.2.826.0.1.3680043.2.1125.1.34918616334750294149839565085991567/1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm, EventId=1010, EventName=DownloadingArtifactFromStorageService, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Downloading artifact 1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm to /var/lib/holoscan/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH/1.2.826.0.1.3680043.2.1125.1.19616861412188316212577695277886020/1.2.826.0.1.3680043.2.1125.1.34918616334750294149839565085991567/1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm. 
mdl-tm        | 2023-10-25 00: 52: 09.7890|1010|DEBUG|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|source=1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm.json, target=/var/lib/holoscan/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH/1.2.826.0.1.3680043.2.1125.1.19616861412188316212577695277886020/1.2.826.0.1.3680043.2.1125.1.34918616334750294149839565085991567/1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm.json, EventId=1010, EventName=DownloadingArtifactFromStorageService, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Downloading artifact 1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm.json to /var/lib/holoscan/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH/1.2.826.0.1.3680043.2.1125.1.19616861412188316212577695277886020/1.2.826.0.1.3680043.2.1125.1.34918616334750294149839565085991567/1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm.json. 
mdl-tm        | 2023-10-25 00: 52: 10.0870|1006|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|hostPath=/home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH, containerPath=/var/holoscan/input/, EventId=1006, EventName=DockerInputMapped, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Input volume mapping host==/home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH, container=/var/holoscan/input/. 
mdl-tm        | 2023-10-25 00: 52: 10.0872|1007|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|hostPath=/home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/outputs/env_HOLOSCAN_OUTPUT_PATH, containerPath=/var/holoscan/output/, EventId=1007, EventName=DockerOutputMapped, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Output volume mapping host==/home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/outputs/env_HOLOSCAN_OUTPUT_PATH, container=/var/holoscan/output/. 
mdl-tm        | 2023-10-25 00: 52: 10.0873|1008|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|key=HOLOSCAN_INPUT_PATH, value=/var/holoscan/input/, EventId=1008, EventName=DockerEnvironmentVariableAdded, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Environment variabled added HOLOSCAN_INPUT_PATH=/var/holoscan/input/. 
mdl-tm        | 2023-10-25 00: 52: 10.0874|1008|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|key=HOLOSCAN_OUTPUT_PATH, value=/var/holoscan/output/, EventId=1008, EventName=DockerEnvironmentVariableAdded, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Environment variabled added HOLOSCAN_OUTPUT_PATH=/var/holoscan/output/. 
mdl-tm        | 2023-10-25 00: 52: 10.0874|1008|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|key=HOLOSCAN_MODEL_PATH, value=/opt/holoscan/models/, EventId=1008, EventName=DockerEnvironmentVariableAdded, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Environment variabled added HOLOSCAN_MODEL_PATH=/opt/holoscan/models/. 
mdl-tm        | 2023-10-25 00: 52: 10.0875|1008|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|key=HOLOSCAN_WORKDIR, value=/var/holoscan/, EventId=1008, EventName=DockerEnvironmentVariableAdded, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Environment variabled added HOLOSCAN_WORKDIR=/var/holoscan/. 
mdl-tm        | 2023-10-25 00: 52: 10.0891|1002|ERROR|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|EventId=1002, EventName=ErrorDeployingContainer, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Error deploying Container. Docker.DotNet.DockerApiException: Docker API responded with status code=BadRequest, response={
    "message": "invalid mount config for type \"bind\": bind source path does not exist: /home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH"
}
mdl-tm        | 
mdl-tm        |    at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
mdl-tm        |    at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
mdl-tm        |    at Docker.DotNet.ContainerOperations.CreateContainerAsync(CreateContainerParameters parameters, CancellationToken cancellationToken)
mdl-tm        |    at Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin.ExecuteTask(CancellationToken cancellationToken) in /app/src/TaskManager/Plug-ins/Docker/DockerPlugin.cs:line 205
mdl-tm        | 2023-10-25 00: 52: 10.0906|106|DEBUG|Monai.Deploy.WorkflowManager.TaskManager.TaskManager|eventType=md.tasks.update, reason=PluginError, EventId=106, EventName=SendingTaskUpdateMessage, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent|Sending md.tasks.update, Status=PluginError . 
mdl-tm        | 2023-10-25 00: 52: 10.0911|10000|INFO|Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessagePublisherService|endpoint=rabbitmq, virtualHost=monaideploy, exchange=monaideploy, topic=md.tasks.update, EventId=10000, EventName=PublshingRabbitMQ, @messageId=6da618d6-ada7-41b5-9427-46a775c7f8ce, @applicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent|Publishing message to rabbitmq/monaideploy. Exchange: monaideploy, Topic: md.tasks.update.
SameerShanbhogue commented 10 months ago

mdl-wm | 2023-10-31 09:18:13.2528|20001|ERROR|Monai.Deploy.Storage.MinIO.MinIoStorageService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|File '1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc/env_MONAI_OUTPUTPATH' could not be found in 'monaideploy'. mdl-wm | 2023-10-31 09:18:13.2567|700000|DEBUG|Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|Artifact Passed data Artifact {"name":"env_MONAI_OUTPUTPATH","value":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc/env_MONAI_OUTPUTPATH","mandatory":true}, Path 1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc/env_MONAI_OUTPUTPATH, ArtifactType Post-Task Output Artifact, Exists False mdl-wm | 2023-10-31 09:18:13.2601|700004|DEBUG|Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|Mandatory output artefacts for task liver are missing. mdl-wm | 2023-10-31 09:18:13.2622|200015|INFO|Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|TaskFailed, Task {"execution_id":"10edc066-56bc-43cf-8cf2-47291ebf3dfc","workflow_instance_id":"6df274a9-49ab-4847-b1dc-69a8154a6612","task_type":"docker","task_start_time":"2023-10-31T09:18:03.181Z","task_end_time":null,"execution_stats":{"IdentityKey":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e","workflowId":"6df274a9-49ab-4847-b1dc-69a8154a6612","duration":"1793.7903","startedAt":"2023-10-31T09:18:08","finishedAt":"2023-10-31T09:18:10"},"task_plugin_arguments":{"container_image":"sameershanbhogue/my_app-x64-workstation-dgpu-linux-amd64:1.0","server_url":"unix:///var/run/docker.sock","entrypoint":"/bin/bash,-c","command":"python3 -u /opt/monai/app/app.py","task_timeout_minutes":"30","temp_storage_container_path":"/var/lib/monai/","env_MONAI_INPUTPATH":"/var/monai/input/","env_MONAI_OUTPUTPATH":"/var/monai/output/","env_MONAI_MODELPATH":"/opt/monai/models/","env_MONAI_WORKDIR":"/var/monai/"},"task_id":"liver","previous_task_id":"router","status":2,"reason":0,"input_artifacts":{"env_MONAI_INPUTPATH":"1918d483-8609-4c64-b622-6413f43f9abb/dcm"},"output_artifacts":{},"output_directory":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc","result":{"JobIdentity":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e"},"next_timeout":"2023-10-31T10:18:03.181Z","timeout_interval":60.0,"acknowledged_task_errors":null}, workflowInstance {"Version":"1.0.0","id":"6df274a9-49ab-4847-b1dc-69a8154a6612","ae_title":"MONAI-DEPLOY","workflow_name":"ai-liver-seg","workflow_id":"abdf887f-126d-4de6-bbb9-3e58a3ba81a8","payload_id":"1918d483-8609-4c64-b622-6413f43f9abb","start_time":"2023-10-31T09:18:02.967Z","status":0,"bucket_id":"monaideploy","input_metadata":{},"tasks":[{"execution_id":"8833f1c4-81c7-43d5-90b5-62a8a065aa15","workflow_instance_id":"6df274a9-49ab-4847-b1dc-69a8154a6612","task_type":"router","task_start_time":"2023-10-31T09:18:02.969Z","task_end_time":"2023-10-31T09:18:03.3Z","execution_stats":{},"task_plugin_arguments":{"workflow_name":"ai-liver-seg"},"task_id":"router","previous_task_id":"","status":3,"reason":0,"input_artifacts":{},"output_artifacts":{},"output_directory":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/8833f1c4-81c7-43d5-90b5-62a8a065aa15","result":{},"next_timeout":"2023-10-31T10:18:02.969Z","timeout_interval":60.0,"acknowledged_task_errors":null},{"execution_id":"10edc066-56bc-43cf-8cf2-47291ebf3dfc","workflow_instance_id":"6df274a9-49ab-4847-b1dc-69a8154a6612","task_type":"docker","task_start_time":"2023-10-31T09:18:03.181Z","task_end_time":null,"execution_stats":{"IdentityKey":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e","workflowId":"6df274a9-49ab-4847-b1dc-69a8154a6612","duration":"1793.7903","startedAt":"2023-10-31T09:18:08","finishedAt":"2023-10-31T09:18:10"},"task_plugin_arguments":{"container_image":"sameershanbhogue/my_app-x64-workstation-dgpu-linux-amd64:1.0","server_url":"unix:///var/run/docker.sock","entrypoint":"/bin/bash,-c","command":"python3 -u /opt/monai/app/app.py","task_timeout_minutes":"30","temp_storage_container_path":"/var/lib/monai/","env_MONAI_INPUTPATH":"/var/monai/input/","env_MONAI_OUTPUTPATH":"/var/monai/output/","env_MONAI_MODELPATH":"/opt/monai/models/","env_MONAI_WORKDIR":"/var/monai/"},"task_id":"liver","previous_task_id":"router","status":2,"reason":0,"input_artifacts":{"env_MONAI_INPUTPATH":"1918d483-8609-4c64-b622-6413f43f9abb/dcm"},"output_artifacts":{},"output_directory":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc","result":{"JobIdentity":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e"},"next_timeout":"2023-10-31T10:18:03.181Z","timeout_interval":60.0,"acknowledged_task_errors":null}],"acknowledged_workflow_errors":null}, patientDetails {"patient_id":"C3N-00198","patient_name":"C3N-00198","patient_sex":"M","patient_dob":null,"patient_age":"055Y","patient_hospital_id":null}, correlationId 15d68444-fa6e-4c17-8fa8-e4318ca0ead7, taskStatus Failed

Is this error related to the bug

mocsharp commented 10 months ago

@SameerShanbhogue did you build your app using MONAI Deploy App SDK 0.5 or 0.6? If 0.6, then it is likely to be related. Thanks.

MMelQin commented 10 months ago

@SameerShanbhogue did you build your app using MONAI Deploy App SDK 0.5 or 0.6? If 0.6, then it is likely to be related. Thanks.

The log entry has the telltale sign that @SameerShanbhogue indeed built and tested the MAP with App SDK v0.6. ''' "task_plugin_arguments":{"container_image":"sameershanbhogue/my_app-x64-workstation-dgpu-linux-amd64:1.0","server_url":"unix:///var/run/docker.sock","entrypoint":"/bin/bash,-c","command":"python3 -u /opt/monai/app/app.py"," '''

SameerShanbhogue commented 10 months ago

@mocsharp @MMelQin Yes v0.6 is there a work around till bug is fixed.

image

mocsharp commented 10 months ago

I have a fix coming soon!

mocsharp commented 10 months ago

@SameerShanbhogue Please try https://github.com/Project-MONAI/monai-deploy/releases/tag/monai-deploy-express-v0.5.0 and let me know if that works for you! Thanks!

MMelQin commented 10 months ago

@mocsharp @MMelQin Yes v0.6 is there a work around till bug is fixed.

image

As @mocsharp mentioned, the fix is known, and pull request already reviewed and merged. Please try MD Express v0.5.0 as mentioned above.

As for workaround with earlier version of MD Express, there is no good ones if the MAP (App SDK v0.6) is built with no-root user, because the Task Manager (and the Docker plugin), when run in container as root, will dynamically create a folder on the host FS to use as the output folder for the MAP container, and the dynamically created folder would not have "Write" permission for other users.

If you build the MAP with container user be root, then it would work. However, the Packager in v0.6 has a little issue where it does not fall back if GUID 0 in the container already exists, and would fail; this issue has also been addressed, but to be released in v1.0.

SameerShanbhogue commented 10 months ago

Thank You @mocsharp @MMelQin

MD Express v0.5.0 with Liver Tumor 2.0 workflow definition works !!