search

bionitio-team / bionitio-python

Demonstrating best practices for bioinformatics command line tools
MIT License
24 stars 12 forks source link

Add pytest tests #10

Open claresloggett opened 4 years ago

claresloggett commented 4 years ago

I'd like to add pytest tests. This is a much more natural and easier-to-maintain way to test Python tools than *nix script-based tests. It's good for both unit and functional tests.

I don't want to do this without checking first though as I can see that tests are shared between bionitio templates and stored in https://github.com/bionitio-team/bionitio-test . I assume this is using submodules.

I'm not sure what the right approach is here. Obviously if the Python repo switches to pytest we lose the submodule-based ability to update all tests at once. But I also think tests using a Python framework are definitely the right approach both for new tool writers, and for long-term maintenance, to the degree that I think it's worth losing this.

Possibly what I could do is keep just the test data from the submodule, and then the pytest tests would need to be kept in sync with any changes to the shared test data.

Any thoughts? My inclination is actually just to copy the current test data over and switch entirely to pytest-based tests - it's cleaner and I would assume bionitio tests aren't going to be updated that often anyway.

claresloggett commented 4 years ago

I've been thinking this for a while but it was just brought home to me when I went to add stuff to @lonsbio's biomojify, which uses bionitio, and realised it doesn't have a test framework in place.. which it would if I'd got round to adding the framework to bionitio earlier!

multimeric commented 4 years ago

So how it works currently is that we have the shared set of tests and test data which, yes, is included as a submodule. This is written in bash to make it language independent etc, and also we use the same tests for each implementation to prove that each implementation has the exact same behaviour, which I think is important.

However,, each repo also has its own unittests, which are our "best practise" recommendation. This repo currently uses unittest (https://github.com/bionitio-team/bionitio-python/blob/master/bionitio/bionitio_test.py), but I agree that converting to pytest would be an improvement. And since these tests are independent of the shared tests, it won't cause us to lose any features.

claresloggett commented 3 years ago

Ah, right, I was looking for a separate tests directory and stopped looking when I saw the functional tests. Ok, I'll have a look at replacing the unittest tests with pytest. If no-one objects I'll put pytest tests in a tests or unit_tests directory so it's on par with the functional_tests directory.

I take your point on having functional tests demonstrate that each implementation has the same behaviour.