Local test execution is too slow.

[jamessynge]

We want folks to be able to run the tests easily and often. Unfortunately a number of them are quite slow, including the new astrohaven test.

[jamessynge]

Here are the 20 slowest tests from a recent run of pytest --durations=20

================== slowest 20 test durations ==================
60.89s call     pocs/tests/test_pocs.py::test_run
40.44s call     pocs/tests/test_pocs.py::test_run_wait_until_safe
31.67s call     pocs/tests/test_pocs.py::test_run_no_targets_and_exit
15.12s call     pocs/tests/test_dispatch_scheduler.py::test_set_observation_then_reset
14.87s call     pocs/tests/test_observatory.py::test_status
11.26s call     pocs/tests/test_observatory.py::test_get_observation
11.21s call     pocs/tests/test_dispatch_scheduler.py::test_observed_list
11.20s call     pocs/tests/test_dispatch_scheduler.py::test_continue_observation
10.12s call     pocs/tests/test_pocs.py::test_run_power_down_interrupt
10.07s call     pocs/tests/test_pocs.py::test_run_interrupt_with_reschedule_of_target
9.14s call     pocs/tests/test_pocs.py::test_unsafe_park
7.83s call     pocs/tests/test_dispatch_scheduler.py::test_new_observation_seq_time
7.03s call     pocs/tests/test_camera.py::test_observation[simulator]
5.01s call     pocs/tests/test_camera.py::test_exposure_collision[simulator]
5.01s call     pocs/tests/test_camera.py::test_exposure[simulator]
4.81s call     pocs/tests/test_observatory.py::test_autofocus_coarse
4.35s call     pocs/tests/test_pocs.py::test_power_down_while_running
3.93s call     pocs/tests/test_camera.py::test_autofocus_coarse[simulator]
3.79s call     pocs/tests/test_dispatch_scheduler.py::test_get_observation
3.79s call     pocs/tests/test_dispatch_scheduler.py::test_reset_observation

[jamessynge]

The situation is getting worse. We need to figure out how to streamline the process. It used to be under 5 minutes to run the tests, but I think test_pocs.py now takes longer than that.

As a start, we need to figure out where the time is going.

[wtgee]

Yes, the addition of PanFileDB made the tests much longe, which was mentioned in #414 as the reason I was hesitant to merge it.

I think we could mark test_pocs.py as a "slow" and only run in during a full testing suite (i.e. on travis or when manually specified), relying somewhat on the other tests actually providing good enough coverage.

As mentioned before, I certainly don't run the full testing suite every time so this is usually not a problem for me. I'll run any specific test I am working on until it works then, run the full suite, then run any tests that fail as individual runs. If you have one specific test that is failing and want to explore it there is no reason to run the full suite.

[wtgee]

Sees like when testing locally we could choose one of either mongo, file, or memory and cut test time in 1/3

[jamessynge]

Agreed. We just need to have the ability to optionally use all three when testing other components, and direct unit tests of each.

On Mon, Aug 27, 2018, 11:11 PM Wilfred Tyler Gee notifications@github.com wrote:

Sees like when testing locally we could choose one of either mongo, file, or memory and cut test time in 1/3

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/panoptes/POCS/issues/191#issuecomment-416436771, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYmU_NFgdK7InY03dE-SO6WecFPYqILks5uVLTigaJpZM4RCFWY .

[wtgee]

As of right now the full testing suite is about 26 minutes (run on travis). This is after removing mongo.

The most time is consumed by test_pocs, test_camera, and test_arduino_io. Some of that can be cleaned up, other parts might be hard.

What we might also want is better documentation for developers about how to run individual tests.