pytest for plot unit tests

[pgkirsch]

To partially address #101

[pgkirsch]

@bqpd @galbramc I'm using pytest-mpl to unit test the plotting functions. Does that mean I need to create an equivalent of gpkit.test.helpers.run_tests that can generate xml from pytest unit tests?

[galbramc]

That would be good. Are there other options for generating files with the results of the tests that can be parsed by JenkinsCI?

[pgkirsch]

Looks like yes! https://docs.pytest.org/en/latest/how-to/output.html?highlight=jenkins

To create result files which can be read by Jenkins or other Continuous integration servers, use this invocation:

pytest --junitxml=path

to create an XML file at path.

All of the GPfit unit tests can now be run with pytest by the way pytest tests/. There are 9 unit tests that use pytest-mpl so I believe the complete command will need to be:

pytest --junitxml=path --mpl

[galbramc]

So do we need to change the scripts for how to run the units tests now? Here is how they are currently executed:

echo "import gpfit.tests; gpfit.tests.run(xmloutput=True)" > test.py
echo "Current working directory:"
ls `pwd`
python $COVERAGE run --source=$WORKSPACE/gpfit --omit=$WORKSPACE/gpfit/tests/* test.py
python $COVERAGE xml
python $COVERAGE html -d $WORKSPACE/htmlcov

# Execute the examples to make sure they work (but they don't contribute to coverage)
for i in `ls $WORKSPACE/docs/source/examples/*.py`; do
  python $i
done

[pgkirsch]

Is $COVERAGE an environment variable to do with coverage.py?

If so, I believe that 4th line can be changed to:

python $COVERAGE run --source=$WORKSPACE/gpfit --omit=$WORKSPACE/gpfit/tests/* pytest --junitxml="test_reports" --mpl

The first 3 lines can then be deleted...I think (unless you want to keep the 2nd/3rd for debugging).

[galbramc]

Sorry this fell off my radar. Yes $COVERAGE refers to coverage.py. The 2nd two lines are processing the coverage information after running the tests with the first line. Can you confirm that running things locally on your machine this works:

python `which coverage` run --source=gpfit --omit=gpfit/tests/* test.py pytest --junitxml="test_reports" --mpl

Whats the reason for moving to pytest btw?

[pgkirsch]

No worries!

That command doesn't run but the following seems to work correctly:

 `which coverage` run -m --source=gpfit --omit=gpfit/tests/* pytest --junitxml="test_reports" --mpl

(need -m but don't need python or test.py)

pytest has a neat extension called pytest-mpl that allows for unit testing of plots generated with matplotlib. It also turns out that pytest can run all unit tests written using python's native unittest module so the migration required barely any changes to the testing code.

[bqpd]

oh sounds like i ought to look into that for gpkit as well

[galbramc]

test this please

[galbramc]

test this please

[galbramc]

Does --junitxml=test_reports generate a directory containing xml files, or does it make one single xml file called "test_reports"?

[pgkirsch]

It seems to create one single xml file

[galbramc]

Does it add the xml extension or should it be --junitxml=test_reports.xml to get the extension?

[pgkirsch]

Ah it should be --junitxml=test_reports.xml

[galbramc]

test this please

[galbramc]

test this please

[galbramc]

Ok the scripts and jenkins are updated to use pytest now, and it seems to be working.

[pgkirsch]

Awesome thank you!

[pgkirsch]

@bqpd (or @whoburg) mind giving this a review when you get a chance? or I can just merge..

[pgkirsch]

test this please

[whoburg]

👏