search

NVIDIA / ai-assisted-annotation-client

Client side integration example source code and libraries for AI-Assisted Annotation SDK
Other
307 stars 64 forks source link

3D slicer Nvidia AIAA local Server error for fully automatic liver segmentation #76

Closed angie1193 closed 3 years ago

angie1193 commented 3 years ago

Run Segmentation for model: segmentation_ct_liver_and_tumor - Status: 404; Response: b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n404 Not Found\n

Not Found

\n

The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

\n'

Is there any local server address in Germany I can manually enter to access the server as leaving it blank does not work. Thank you.

SachidanandAlle commented 3 years ago

We recommend to install your own NVIDIA AIAA Server to run auto-segmentations and other features provided.

https://docs.nvidia.com/clara/tlt-mi/aiaa/installation.html

lassoan commented 3 years ago

Based on our experience, most users is not willing to put more than a few minutes effort for first evaluating of a segmentation tool. We can reach hundreds times more users if we offer a demo server, which is crucial if we want Clara to gain developer and user mindshare faster than competitors.

I can reproduce the reported problem and most likely it is because the current AIAA client is not compatible with v2 server anymore. We tried to switch to the latest server, but it proved to be too unstable on our configuration (see https://github.com/NVIDIA/ai-assisted-annotation-client/issues/62).

I would recommend to close this issue if either v2 server support is officially removed, or the reported error is fixed.

lassoan commented 3 years ago

Users keep running into this issue (see for example here).

The current client is not compatible anymore with the v2 server but we cannot upgrade the default server to the latest version until it is as robust as the v2 version (see https://github.com/NVIDIA/ai-assisted-annotation-client/issues/62).

jamesobutler commented 3 years ago

Another user report of the issue: https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/56

lassoan commented 3 years ago

Updated link to user report: https://discourse.slicer.org/t/nvidia-aiaa-segment-editor-effect-returns-url-not-found-error/16811

We would like users to have positive experience with the NVidia AIAA tool, but we cannot provide our own demo server due to backward incompatibility of the current client with server v2; and instability of server v3. There is no public server provided by NVidia or other groups either.

@SachidanandAlle please advise what we should do.

SachidanandAlle commented 3 years ago

As you have loaded multiple models in a small server. I am not sure if the issue is due to stability of 3.0 version (which I haven't so far heard from any other customers)... or due to larger number of models in the server. Sepcially some of the models take more GPU resource and when multiple users accessing, there might be a good chance of getting resources exausted.

How many more models are loaded in 3.0 vs 2.0 environment..?

SachidanandAlle commented 3 years ago

Also adding Prerna (NVIDIA PM)... @pdogra89

lassoan commented 3 years ago

Thank you for looking into this.

As you have loaded multiple models in a small server. I am not sure if the issue is due to stability of 3.0 version (which I haven't so far heard from any other customers)... or due to larger number of models in the server.

It may be a low-resource situation, but a server of any size can run into such situation (it is just a matter of number and size of models and users). Note that the server can run complete the segmentation just fine. A few times. But after a few successful runs (one after the other, by a single user), the server fails, and it fails persistently - it never recovers. It is tolerable if the entire container crashes, because we can then automatically restart it. It is also fine if the server automatically detects and fixes the error internally. It just the internal persistent error that prevents us from using the server (from outside it looks that the server is still OK, but its internal state is broken). See details in #62.

It would be also nice to clarify if the latest client is supposed to be compatible with server 2.0. If it should be compatible then there is a bug that needs to be fixed (see details above). If it is expected that the current client is not backward compatible with a 2.0 version server then a meaningful error message should be reported instead of a generic "404 - not found".

SachidanandAlle commented 3 years ago

One possible solution (Not guranteed) could be... Allow single access. For example, try to set the threads to 1 for all V1 Api access in AIAA.conf (for apache). This file can be located in

/etc/apache2/sites-available/AIAA.conf

WSGIDaemonProcess AIAA_V1 threads=1 python-path=/opt/nvidia/medical

lassoan commented 3 years ago

Thank you, I'll try this. I'll also try server 3.1.

lassoan commented 3 years ago

We'll update the default Slicer server within a few weeks. Hopefully with the new hardware and latest Clara server everything will work robustly.

lassoan commented 3 years ago

The default 3D Slicer segmentation server has been upgraded to Clara 4.0 and it supports all segmentation modes (auto-segmentation, segmentation from boundary points, and deepgrow), with many new models. It will be available the day after https://github.com/NVIDIA/ai-assisted-annotation-client/pull/88 is merged.

@SachidanandAlle with this new server and latest software stack everything seems to work robustly. Thanks for all your help with the investigations.