As a MLOps engineer I would like to perform automated tests to test the functional behavior of the container with the trained model.

opendatahub-io / ai-edge

ODH integration with AI at the Edge usecases

Apache License 2.0

11 stars 19 forks source link

As a MLOps engineer I would like to perform automated tests to test the functional behavior of the container with the trained model. #5

Open LaVLaS opened 1 year ago

LaVLaS commented 1 year ago

Requirements:

These tests should be standalone tests that can be included as a verification step in build and push pipeline.
These tests should be a simple smoke tests
Document the steps to verify functionality of the running model

Sub Tasks

[ ] Proposal for implementing the testing feature

gzaronikas commented 1 year ago

Can we elaborate a bit on the what the testing would look like? Are we expecting functional/integration tests? Things to consider:

The pipeline where the tests live and executed
Minimal scope of testing. Should the test run an instance of the build image and send it a simple known inference request and verify the response?
Functional tests vs Integration tests

piotrpdev commented 1 year ago

Stefan's code does this, see #16 for more details.

LaVLaS commented 1 year ago

Can we elaborate a bit on the what the testing would look like? Are we expecting functional/integration tests?

The goal is to show that we can verify basic functionality of any example model. A is not required but we should be able to run some test that can determine:

if the model image was built successfully
the model server is online
the model is running correct by confirming basic functionality

devguyio commented 1 year ago

For the PoC this is covered by #2

devguyio commented 1 year ago

I believe we discovered that what was intended by this is to deploy the model in a test env as part of the pipeline. This is not covered by #2

StevenTobin commented 1 year ago

Some things that should be included to call testing of the image/testing in PoC complete

Tag a cluster as the stage environment in ACM.
Make it clear in the pipeline where the built image would be expected to be deployed to a stage environment.
Make it clear where the gates would be - built image, smoke tests -> deploy to stage, more in-depth tests -> deploy to prod.
We could build some metrics around these, include a dashboard that identifies how many images made it through stage etc.?

adelton commented 1 year ago

So the test/stage environment use-case is tracked in https://github.com/opendatahub-io/ai-edge/issues/59.

If we factor that out, does the current approach with the test-mlflow-image Pipeline already cover the rest of this issue today?

piotrpdev commented 1 year ago

I'm down to close this issue and focus on #59, this issue was meant for simple tests anyway and it would be good to close it since it's part of phase 1. @devguyio @StevenTobin @LaVLaS ?