Adding automated test suite mechanism to branch #5

[EiffL]

This PR proposes a mechanism to automate the testing against GalSim. Here is the idea:

• The GalSim repo is included as a git submodule inside the test folder. This corresponds to a specific version that we want to test against, typically the latest released version.

• In a yaml file, galsim_tests_config.yaml we can specify which galsim test modules we want to run, and we can specify what error messages to ignore. the file looks like this:

  
  # This defines which tests from the GalSim suite will run
  enabled_tests:
  - test_gaussian.py
  - test_exponential.py

This documents which error messages will be allowed

without being reported as an error. These typically

correspond to features that are not implemented yet

in jax_galsim

allowed_failures:

• "module 'jax_galsim' has no attribute 'fits'"
• "module 'jax_galsim' has no attribute 'PositionD'"
• "module 'jax_galsim' has no attribute 'BaseDeviate"
• "unsupported operand type(s) for *="

  
  which means that if an error occurs during a test, and that error has to do with the fact that fits file are not supported yet, we don't fail the test, we will skip it.

• With some pytest magic in the conftest.py file, we tell pytest to:
  1. only run the enabled tests and skip the rest of the GalSim test suite
  2. replace the import galsim at the top of the test files by import jax_galsim as galsim
  3. if an error arises corresponding to an "allowed failure" mode, just skip it for now.

and here is what running the tests look like:

[francois@Achernar JAX-GalSim]$ pytest
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss-----.sssss [ 39%]
s-----.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 78%]
sssssssssssssssssssssssssssssssssssssssssssssssssss---.---sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss                                                                                                                    [100%]
========================================================================================================================== short test summary info ===========================================================================================================================
3 passed, 652 skipped, 16 allowed failure in 1.43s

[ismael-mendoza]

Awesome! Never knew that pytest was this flexible :D

[b-remy]

Since we are using the GalSim repo as a git submodule, we might want to write somewhere how to install or clone the repo so we can run the tests. I imagine that once this branch will be merged into the main branch, we will just clone and install with

git clone --recurse-submodules https://github.com/GalSim-developers/JAX-GalSim
cd JAX-GalSim
pip install .

And add pytest to the requirements in the setup.py

[EiffL]

Thank you @b-remy ! I've added the instructions to clone the repo including submodules in the contibuting.md being developed here: https://github.com/GalSim-developers/JAX-GalSim/blob/contributing/CONTRIBUTING.md

And, I've added pytest to the dependencies in the setup.py

[EiffL]

because this is not a PR for the main branch I'll go ahead and merge these modifs into #7 given your feedback