[WS-2] Improve coverage with unit and integration tests

alexmorley / pykilosort

[WIP] Python port of Kilosort 2

GNU General Public License v2.0

0 stars 0 forks source link

[WS-2] Improve coverage with unit and integration tests #2

Open alexmorley opened 4 years ago

alexmorley commented 4 years ago

Requires:

[X] Setting up continuous integration
[ ] Add a whole bunch more tests. Aim for codecov at 90% (only locally as GPU can't be tested in the cloud cheaply for now) and integrated with CI
[x] Solve the "Need data for tests" problem
[X] Split the unit tests into GPU and CPU. (See below)
[ ] :question: Proposal: Ask circleCI nicely to give us a GPU machine for testing. (See below)

alexmorley commented 4 years ago

CI services don't offer GPU machines*

*within a reasonable budget

Biggest problem here is that the only CI service that offer GPU machines is circleCI which are very expensive (CPU is free). Potential solutions:

Split the unit tests into GPU and CPU. GPU ones will just have to be run locally for now. We can mock out parts of the algorithm that use the GPU so this is probably the neatest short term but prevents us from running integration tests.
Ask circleCI nicely to give us a GPU machine for testing. They do do this for open source projects (at least they say they do) so this is worth a shot.
Just run all testing locally as it is now. Increases the burden for reviewing new code and reduces confidence we aren't breaking things.

alexmorley commented 4 years ago

Need data for tests

Lots of the things we want to test need data. There are three solutions that we can use (we will probably need all of them).

Integration Tests: It's best to use some reasonably sized data here (and real data ideally). So I think we should host some data (or grab from somewhere with an API) in order to run the tests.
Unit Tests I: For some unit tests we want to specifically check that the output hasn't changed. For these, we can either include the input/expected_output in the repo (if they are very small) or we can download them like for the integration tests (probably better).
Unit Tests II: For others, we can just use some very simple simulations e.g. we generate the inputs and expected output properties on the fly.

alexmorley commented 4 years ago

Integration Tests: It's best to use some reasonably sized data here (and real data ideally). So I think we should host some data (or grab from somewhere with an API) in order to run the tests.

Solution in #8. Potential drawback is dependency on spikeforest. Maybe should zip it and host it somewhere too.