Machine learning provider interface

[Rui-Jesus]

Summary

This is a work in progress to implement a new interface for image analysis providers in Dicoogle. The endgoal is to provide a standard interface, compatible with DICOM to manage image analysis providers and algorithms. This is particularly important in Pathology, given the heterogeneity in the field. Another important consideration with this interface is the possibility to integrate services like Google AI, that exposes AutoML interfaces, allowing users with little IT knowledge to develop their own inference models.

Here is a draft of the intended architecture: The basic idea is to provide a standard interface that developers can use to implement image analysis providers. These providers should allow the creation and management of inference models, i.e, creation of labels, labelled datasets, and inference requests.

All the methods from the providers are exposed via servlets located in Dicoogle core. The API definitions took inspiration from MONAI and the DICOM Standards for image segmentation and overlays.

This PR should be seen as the initial step in supporting AI integrations in Dicoogle. It currently implements the following endpoints/functionalities:

• Inference requests, for DICOM objects or image annotations in WSI
• Training requests
• Cancel training
• List models

Additionally, the following miscellaneous features were implemented:

• A new flag "wsi" to indicate whether or not this dicoogle installation supports WSI. This is a temporary bandaid while the problems with supporting both WSI and radiology at the same time are not fixed (there is a conflict in the codecs)
  • When set to true, a image reader for DICOM is registered, needed to work with WSI images
• A new "roi" endpoint to extract ROIs (regions of interest) from WSIs. It is described in the api documentation.

Future Work

• Implementation of the remaining endpoints, planned in the MLProviderInterface:
  • Create model endpoint
  • Delete model endpoint
• Support for other types of data such as CSV
• Support of segmentation and classification models in the dataset object

Tasks to be implemented in this PR

• [x] Inference requests
• [x] Model listing
• [x] Datastore of labels/annotations (for training)
  • Should be sufficiently flexible to support classification and segmentation tasks.
  • Dataset parameters should be somewhat configurable (patch size, mask or contours, etc.)
• [x] Caching of images (used to preemptively upload data to a provider to run inference on, needed for example in MONAI
• [x] Implementation of interface method that returns the implemented endpoints of each provider.
• [x] Integration of the bulk annotation supplement in annotation definition

Resources

• Workgroup 23 - Application Hosting and Artificial Intelligence
• Bulk Annotation.
• Overlays
• DICOM Segmentation

Test plan

• Test this contribution with a plugin that implements this interface
• Test the several endpoints and validate the responses
  • Documentation of the new endpoints added can be found in the openapi documentation
  • Validate the documentation too.

[bastiao]

Please resolve the conflicts in this PR. As we previously discussed, we have reached a significant milestone, and it is now appropriate to review the API definition and proceed with merging this pull request.

[joselfrias]

Note: dim-providers must be configured in server.xml, otherwise the server fails to launch.

[bastiao]

Looks good! And of course, it works as we have been using with it internally. @Rui-Jesus question about MLCSVDataset: is this used by any use case , we are currently handling? I'm wondering if we really have any advantage of calling it to MLCSVDataset or having something more generic.

I will keep diving on the API to hopefully merge this one soon.

[Rui-Jesus]

Looks good! And of course, it works as we have been using with it internally. @Rui-Jesus question about MLCSVDataset: is this used by any use case , we are currently handling? I'm wondering if we really have any advantage of calling it to MLCSVDataset or having something more generic.

I will keep diving on the API to hopefully merge this one soon.

I'm open to any name change, on any of the data objects or even the interface. MLCSVDataset is not being used anywhere currently. I entertained the idea of supporting tabular datasets in this API and thought of using a CSV file for it, for a future where we would need to work with tabular data.

I'm not sure how this could be accomplished directly from viewer applications, but maybe for third party services that can compute tabular data it could be useful. But I do not have any plans to implement it soon.

[Rui-Jesus]

I think this PR is ready for a code-freeze. There are still missing features, but this is an ongoing work, and the API is likely to change.