Evaluate turning jupyter-notebooks into an automated test suite for platform

[ca-scribner]

To aid in automated testing of the platform, can we change some or all of the repo into an automated test suite? Could be run by papermill or some other notebook runner, maybe require a specific input/output format to dictate what "success" is, etc.

[ca-scribner]

Objective

It would be beneficial if we could use some or all of our existing tutorial jupyter-notebooks as automated tests. This would:

• provide us with a number of test cases to demonstrate the platform is working as expected
• help with maintenance of our tutorials, which as we develop new features often become out of date (eg: tutorials on how to access minio)

If not able to easily use existing guides as tests, defining a format for writing future guides so that they could be used as tests would also be helpful. Examples of this are from fastai who built their own notebook runner and use all guide notebooks as automated tests, including assertions, etc.

Research

• Good discussion of automated testing options
  • They mention how fastai and parcels both of advanced test suites that are run over their example and documentation notebooks

Known challenges

KFP

Connections to KFP runner

Many tests involve submitting kubeflow pipelines jobs, which requires a connection to the KFP runner (typically via instantiating a Client()). From inside platform, this can be done simply by calling Client() and using default in-platform authentication. For automated tests handled by a github runner, this would not work.

Possible solutions could be:

• Adding full authentication to Client() calls (see mlops repo's CI for example of off-platform authentication for submitting kfp jobs
  • This requires us changing all Client() calls in all test suite examples to use the off-platform authentication. Some ways to do this could be adding a get_client() helper that is smart enough to fail to off-platform methods, or use something like injecting a mocked Client() into all examples during testing which overrides default behaviour with the off-platform authentication. For a regular suite of .py files using pytest, we can inject this code using the conftest.py file to create a fixture that mocks over Client(), but the below notebook extensions to pytest don't seem to use the code from conftest.py
• Building our test suite to run as a kubeflow pipeline job. This way we could submit from github as a single "run test suite" job (or a series of "run this specific test" jobs), and testing will occur natively on platform. We could run each test against a matrix of notebook server images in order to test the different environments. We might need to mock up any additional function coming from the start-custom.sh that we usually start the servers with.

Tools available

• pytest
  • Not great for native use, but extended by tools below
• pytest extension nvbal
  • lets you detect/run notebooks as tests. Regular pytest --nbval compares cell output to existing output, pytest --nbval-lax just makes sure cells run without error
• fastai's nbdev notebook runner they use for their own testing

[ca-scribner]

So far tried using pytest w/nbdev to run our code as a test suite on a github action runner. The only blocker is the one summarized above with connections to KFP runner. If the goal is to run test cases without any changes to the existing code, I hit trouble with connecting to kfp runner using Client(). Client() needs additional authentication when off platform. If the test suite was entirely .py files, I can use conftest.py (which pytest will run before testing) to make a global mock of the Client() that uses an existing Client fixture that's been properly authenticated off-platform. It works great with .py, but the pytest extensions that enable notebook usage are not affected by the fixtures built in conftest.py (filed issue here to ask about this). Other options are

• editing all the tests to have a more general Client() call, maybe building a helper function that does this automatically (could first try Client() but default back to reading extra creds from environment variables and trying a different way?)
• injecting code to mock the Client() for each test somehow else (some pytest nb extensions had this feature)
• running tests on-platform through a pipeline job rather than off-platform on a github runner

[ca-scribner]

task paused to do other work