Test applications for validating systems that use MONAI Deploy components

[laurencejackson]

Hi all, as discussed in the MONAI deploy WG meeting 7-Apr-22 I think there would be value in creating a suite of MONAI application packages for testing the capabilities of a system that uses MONAI deploy components. These applications would serve the same purpose for MONAI deploy platforms as phantoms do for QA/acceptance testing in imaging.

To give a bit of background: at GSTT we will be deploying the AIDE platform, which is built using the MONAI informatics gateway and MONAI workflow manager components, we will also use the MONAI application runner plugin for the WFM. When AIDE is deployed at GSTT we will need to perform some kind of acceptance testing to ensure that the entire system is fit and safe to use in a clinical setting. When I refer to the system here, I mean the specific combination of software, hardware, network and other local factors that make up the local environment. The testing would aim to verify that this system as a whole can operate reliably under pressure. The testing we perform will aim to explore (non-exhaustive):

• System loading/stress tests: max number of applications that the system (local software + hardware combination) can support. This would probably use many instances of several different test MAPs (small, medium, large resource use)
• Performance metrics of applications when system is running at low-high loads. Metrics including things like startup time, processing time delta, etc
• understand modes of failure

I'm not sure exactly what operations the MAPs themselves would perform, but we would need to test the CPU and GPU resources in isolation and in combination.

I'd like to start a discussion here to gauge whether the community feels these applications would be valuable as part of MONAI and whether there is anything missing from the concept. Please leave your thoughts below!

[dbericat]

Potential datasets talked about today during weekly WG call:

https://datacommons.cancer.gov/data#key-datasets

[dbericat]

I love the idea. Let's take it to the next level and define those small, medium and large MAPs.

We can use the Operating environments guideline to for each of the envs type we are targetting, come up with some numbers.

What do others think? @CPBridge @GreySeaWolf @aylward @ericspod @evberrypi @Danulf @nakfour

[dbericat]

@JoeBatt1989 please add thoughts to this from your performance plan and add it to a roadmap we can track and act upon.

[JoeBatt1989]

@dbericat thanks David, I have added a new ticket in the MWM repo for refinement https://github.com/Project-MONAI/monai-deploy-workflow-manager/issues/118. The sizes in this table are indicative of what we plan to create, although the system specifications and run times should be configurable so we can update as we see fit.

Name | CPU Cores | RAM | GPU | Disk Space -- | -- | -- | -- | -- Dummy (mean pixel calc) |   |   |   |   Small | 2 | 1GB | 1GB | 2GB Medium | 8 | 10GB | 6GB | 15GB Large | 12 | 16GB | 12GB | 25GB