search

Project-MONAI / monai-deploy

MONAI Deploy aims to become the de-facto standard for developing, packaging, testing, deploying and running medical AI applications in clinical production.
Apache License 2.0
98 stars 22 forks source link

Minio Storage Service error #138

Closed SameerShanbhogue closed 11 months ago

SameerShanbhogue commented 1 year ago

Monai Deploy Express work successfully if only 1 dcm file is send from ORTHANC and segmentation is generated. But if a complete series is presented gives following error, no final segmentation is generated. Tried both liver and lung app.

2023-08-01T11:54:31.778893364Z 2023-08-01 11:54:31.7721|20009|ERROR|Monai.Deploy.Storage.MinIO.MinIoStorageService||Storage service error. System.Threading.Tasks.TaskCanceledException: The operation was canceled. mdl-ig | 2023-08-01T11:54:31.778946438Z at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) mdl-ig | 2023-08-01T11:54:31.779212493Z at Minio.MinioClient.ExecuteTaskCoreAsync(IEnumerable1 errorHandlers, HttpRequestMessageBuilder requestMessageBuilder, CancellationToken cancellationToken, Boolean isSts) in /root/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 602 mdl-ig | 2023-08-01T11:54:31.779268984Z at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) mdl-ig | 2023-08-01T11:54:31.779295353Z at Minio.MinioClient.ExecuteTaskCoreAsync(IEnumerable1 errorHandlers, HttpRequestMessageBuilder requestMessageBuilder, CancellationToken cancellationToken, Boolean isSts) in /root/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 602

mocsharp commented 1 year ago

@SameerShanbhogue , what version of MDE are you using? It's usually the opposite, an incomplete series would fail to generate segmentation.

Could you please collect the logs for us to investigate? Thanks!

docker logs mdl-ig> mdl-ig.txt
docker logs mdl-wm > mdl-wm.txt
docker logs mdl-tm > mdl-tm.txt
SameerShanbhogue commented 1 year ago

@mocsharp Hi, the issue is resolved. I was using HDD on AWS for storing images. After moving images to SSD, the issue is resolved, also increased task_timeout_minutes to 60 mins as per advice in https://github.com/Project-MONAI/monai-deploy/issues/137.

Thanks for response

MMelQin commented 11 months ago

@SameerShanbhogue Great to hear that issue was resolved. Yeah, the task timeout modification is critical with running inference applications without visible GPU and limited number of CPU cores. I will mark this as closed.