Closed MMelQin closed 10 months ago
mdl-wm | 2023-10-31 09:18:13.2528|20001|ERROR|Monai.Deploy.Storage.MinIO.MinIoStorageService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary
2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|File '1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc/env_MONAI_OUTPUTPATH' could not be found in 'monaideploy'. mdl-wm | 2023-10-31 09:18:13.2567|700000|DEBUG|Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary
2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary
2[System.String,System.Object]|Artifact Passed data Artifact {"name":"env_MONAI_OUTPUTPATH","value":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc/env_MONAI_OUTPUTPATH","mandatory":true}, Path 1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc/env_MONAI_OUTPUTPATH, ArtifactType Post-Task Output Artifact, Exists False
mdl-wm | 2023-10-31 09:18:13.2601|700004|DEBUG|Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary
2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|Mandatory output artefacts for task liver are missing. mdl-wm | 2023-10-31 09:18:13.2622|200015|INFO|Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary
2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary
2[System.String,System.Object]|TaskFailed, Task {"execution_id":"10edc066-56bc-43cf-8cf2-47291ebf3dfc","workflow_instance_id":"6df274a9-49ab-4847-b1dc-69a8154a6612","task_type":"docker","task_start_time":"2023-10-31T09:18:03.181Z","task_end_time":null,"execution_stats":{"IdentityKey":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e","workflowId":"6df274a9-49ab-4847-b1dc-69a8154a6612","duration":"1793.7903","startedAt":"2023-10-31T09:18:08","finishedAt":"2023-10-31T09:18:10"},"task_plugin_arguments":{"container_image":"sameershanbhogue/my_app-x64-workstation-dgpu-linux-amd64:1.0","server_url":"unix:///var/run/docker.sock","entrypoint":"/bin/bash,-c","command":"python3 -u /opt/monai/app/app.py","task_timeout_minutes":"30","temp_storage_container_path":"/var/lib/monai/","env_MONAI_INPUTPATH":"/var/monai/input/","env_MONAI_OUTPUTPATH":"/var/monai/output/","env_MONAI_MODELPATH":"/opt/monai/models/","env_MONAI_WORKDIR":"/var/monai/"},"task_id":"liver","previous_task_id":"router","status":2,"reason":0,"input_artifacts":{"env_MONAI_INPUTPATH":"1918d483-8609-4c64-b622-6413f43f9abb/dcm"},"output_artifacts":{},"output_directory":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc","result":{"JobIdentity":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e"},"next_timeout":"2023-10-31T10:18:03.181Z","timeout_interval":60.0,"acknowledged_task_errors":null}, workflowInstance {"Version":"1.0.0","id":"6df274a9-49ab-4847-b1dc-69a8154a6612","ae_title":"MONAI-DEPLOY","workflow_name":"ai-liver-seg","workflow_id":"abdf887f-126d-4de6-bbb9-3e58a3ba81a8","payload_id":"1918d483-8609-4c64-b622-6413f43f9abb","start_time":"2023-10-31T09:18:02.967Z","status":0,"bucket_id":"monaideploy","input_metadata":{},"tasks":[{"execution_id":"8833f1c4-81c7-43d5-90b5-62a8a065aa15","workflow_instance_id":"6df274a9-49ab-4847-b1dc-69a8154a6612","task_type":"router","task_start_time":"2023-10-31T09:18:02.969Z","task_end_time":"2023-10-31T09:18:03.3Z","execution_stats":{},"task_plugin_arguments":{"workflow_name":"ai-liver-seg"},"task_id":"router","previous_task_id":"","status":3,"reason":0,"input_artifacts":{},"output_artifacts":{},"output_directory":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/8833f1c4-81c7-43d5-90b5-62a8a065aa15","result":{},"next_timeout":"2023-10-31T10:18:02.969Z","timeout_interval":60.0,"acknowledged_task_errors":null},{"execution_id":"10edc066-56bc-43cf-8cf2-47291ebf3dfc","workflow_instance_id":"6df274a9-49ab-4847-b1dc-69a8154a6612","task_type":"docker","task_start_time":"2023-10-31T09:18:03.181Z","task_end_time":null,"execution_stats":{"IdentityKey":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e","workflowId":"6df274a9-49ab-4847-b1dc-69a8154a6612","duration":"1793.7903","startedAt":"2023-10-31T09:18:08","finishedAt":"2023-10-31T09:18:10"},"task_plugin_arguments":{"container_image":"sameershanbhogue/my_app-x64-workstation-dgpu-linux-amd64:1.0","server_url":"unix:///var/run/docker.sock","entrypoint":"/bin/bash,-c","command":"python3 -u /opt/monai/app/app.py","task_timeout_minutes":"30","temp_storage_container_path":"/var/lib/monai/","env_MONAI_INPUTPATH":"/var/monai/input/","env_MONAI_OUTPUTPATH":"/var/monai/output/","env_MONAI_MODELPATH":"/opt/monai/models/","env_MONAI_WORKDIR":"/var/monai/"},"task_id":"liver","previous_task_id":"router","status":2,"reason":0,"input_artifacts":{"env_MONAI_INPUTPATH":"1918d483-8609-4c64-b622-6413f43f9abb/dcm"},"output_artifacts":{},"output_directory":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc","result":{"JobIdentity":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e"},"next_timeout":"2023-10-31T10:18:03.181Z","timeout_interval":60.0,"acknowledged_task_errors":null}],"acknowledged_workflow_errors":null}, patientDetails {"patient_id":"C3N-00198","patient_name":"C3N-00198","patient_sex":"M","patient_dob":null,"patient_age":"055Y","patient_hospital_id":null}, correlationId 15d68444-fa6e-4c17-8fa8-e4318ca0ead7, taskStatus Failed
Is this error related to the bug
@SameerShanbhogue did you build your app using MONAI Deploy App SDK 0.5 or 0.6? If 0.6, then it is likely to be related. Thanks.
@SameerShanbhogue did you build your app using MONAI Deploy App SDK 0.5 or 0.6? If 0.6, then it is likely to be related. Thanks.
The log entry has the telltale sign that @SameerShanbhogue indeed built and tested the MAP with App SDK v0.6. ''' "task_plugin_arguments":{"container_image":"sameershanbhogue/my_app-x64-workstation-dgpu-linux-amd64:1.0","server_url":"unix:///var/run/docker.sock","entrypoint":"/bin/bash,-c","command":"python3 -u /opt/monai/app/app.py"," '''
@mocsharp @MMelQin Yes v0.6 is there a work around till bug is fixed.
I have a fix coming soon!
@SameerShanbhogue Please try https://github.com/Project-MONAI/monai-deploy/releases/tag/monai-deploy-express-v0.5.0 and let me know if that works for you! Thanks!
@mocsharp @MMelQin Yes v0.6 is there a work around till bug is fixed.
As @mocsharp mentioned, the fix is known, and pull request already reviewed and merged. Please try MD Express v0.5.0 as mentioned above.
As for workaround with earlier version of MD Express, there is no good ones if the MAP (App SDK v0.6) is built with no-root user, because the Task Manager (and the Docker plugin), when run in container as root, will dynamically create a folder on the host FS to use as the output folder for the MAP container, and the dynamically created folder would not have "Write" permission for other users.
If you build the MAP with container user be root, then it would work. However, the Packager in v0.6 has a little issue where it does not fall back if GUID 0 in the container already exists, and would fail; this issue has also been addressed, but to be released in v1.0.
Thank You @mocsharp @MMelQin
MD Express v0.5.0 with Liver Tumor 2.0 workflow definition works !!
Observed: Configured a workflow to run an example map built with MONAI Deploy App SDK v0.6, e.g.,
ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0
, but workflow instance failed with "no output"Expected: MD Express should support the MAPs built with App SDK v0.6 and after; these MAPs are built differently than those with MONAI Deploy App SDK v0.5.x, and have additional requirements on the permission on the host folders (which are to be mapped as MAP output"
Preliminary Analysis: Root cause is in the
writer
permission, or the lack thereof, on the output folder in the host file system.Workaround attempted but did not working due to TM/Docker behavior:
MD Express TaskManager container itself maps host volume to its internal folder, and also uses them for MAP input and output, so tried to first create the host folder with the logged-on user's security context, e.g.
mkdir -p .md/mdtm
. However, when the TaskManager creates further sub-folder for a triggered task, the folder is created as owned byroot
, without sufficient permission for the MAP to write output to.Step to reproduce: Create an workflow definition as below (note, the attributes starting with
env_
do not matter as the actual values are all correct and the MAP has default env vars. However, different error happened, see Additional Info) Download the MD Express,docker compose up
Register the workflow, per instruction on MD Express Readme Upload the a CT abdomen series, per MD Express Readme Send the series to DICOM device destination, monai-deploy Examine the console output ofdocker compose
Additional Info: Used a config that replaces all monai with holoscan, but encountered a different error. This one is because that in the
docker-compose.yml
file, the TaskManager container itself has the volume mapping set to its own/var/lib/monai/
, so the workflow def cannot use"temp_storage_container_path": "/var/lib/holoscan/"
So, stick with
"temp_storage_container_path": "/var/lib/monai/"
, and change it in the following config, to avoid this other binding error.But got a different error from task manager on binding the volumes