usnistgov / dioptra

Test Software for the Characterization of AI Technologies
https://pages.nist.gov/dioptra/
Other
210 stars 30 forks source link

End-to-end hello-world job system test #585

Open keithmanville opened 1 month ago

keithmanville commented 1 month ago

This test should use the Dioptra python client to levearage the v1 REST API. It will submit a simple "hello-world" plugin and entrypoint. The purpose of this test is to confirm the message passing works through the execution of a job and retrieving results.

Assume containers are built and the dioptra python library has passed all unit tests.

  1. Create a dioptra deployment using cruft with no user prompts
  2. Step through a series of REST API calls using the client to

    1. register as user
    2. register a queue
    3. register plugin parameter types
    4. register a plugin (and plugin files)
    5. create an entrypoint using the plugin tasks
    6. register an experiment with an attached entrypoint
    7. submit job to experiment using entrypoint
    8. poll API until job completes
    9. fetch job artifacts / metrics and compare against expectation
    10. teardown and cleanup

    Definition of Done:

    • [ ] End-to-end tests pass in GitHub actions
    • [ ] The feature is merged into dev