search

jupyterhub / repo2docker-action

A GitHub action to build data science environment images with repo2docker and push them to registries.
MIT License
140 stars 27 forks source link

Support testing built image with notebooks and .py files #83

Closed yuvipanda closed 2 years ago

yuvipanda commented 2 years ago

You can automatically test your built image to make sure package additions or removals do not break your code, allowing you to make changes with confidence. pytest is used to run the tests, and pytest-notebook is used to run any Jupyter Notebooks as tests.

  1. Create a directory named image-tests/ in your GitHub repository.
  2. Any .py files you add inside this directory will be discovered and run with pytest inside the built image after the image has successfully built.
  3. Any Jupyter Notebook (.ipynb) files inside this directory will be run with pytest-notebook, and the notebook is considered to have failed if the outputs of the code execution do not match the outputs already in the notebook. A nice diff of the outputs is shown if they differ. See the pytest-notebook docs for more information.
  4. Optionally, a requirements.txt file inside the image-tests/ directory can list additional libraries installed just for the test.

See https://github.com/2i2c-org/utexas-image/runs/5431256062?check_suite_focus=true for an example of a notebook test that fails.

yuvipanda commented 2 years ago

https://github.com/2i2c-org/utexas-image/pull/3 is what I was testing with.

manics commented 2 years ago

This looks cool! Could we run a test in this repo too, since there's a notebook in https://github.com/jupyterhub/repo2docker-action/tree/master/notebooks ?

hamelsmu commented 2 years ago

This is super cool and innovative!!!LGTM

yuvipanda commented 2 years ago

Thank you for the kind words and suggestions, @hamelsmu and @choldgraf! I've accepted those :)

@manics is this repo already built with the same action? I'm actually not sure at all what the notebook here is used for or how I'd do what you're suggesting.

choldgraf commented 2 years ago

Does this action support the ability to look for a particular folder for the environment files to build an image from? If that were the case then we could have our own test/ folder with a little set of repo2docker files that we could use for basic testing.

yuvipanda commented 2 years ago

@choldgraf i don't think it does. I do want to add that support though, but as a different PR.

manics commented 2 years ago

I agree, if you're planning to add tests later anyway then merge this

yuvipanda commented 2 years ago

@manics yeah, I'm actually working on moving some of this to python that will help us add more tests!