Create two-stage unit test

[kylepjohnson]

Coming from offline suggestion by Todd, to solve the problem of our 50-min build times.

• Have two test suites: A fast one for development and a slow one that checks everything. The fast one would be enough for checking most PRs. We could probably leverage Travis's build matrix, so that if the faster of the two suites finishes green, then we can merge with some confidence.
• Travis's cron jobs might be a handy way to check for upstream breakage, too. Just running once a day in the background, only sending alerts if a problem arises.

Question: How about only the doctests be called for the fast build, then the full suite (in src/cltk/tests/) be run for the long build?

Question: We can move the tests dir out of the source code. Should we?

[todd-cook]

One approach:

1. Turn off the doctest switches
2. Create a Unittest class which manually invokes doctests on classes
3. Implement an environment flag check, e.g. "TEST_ALL", and check and skip all but essential tests if not present.
4. Use the env flag check to skip/pass other tests that aren't essential and are not in doctests.
5. The cron job could set the env flag and run with the doctest switch, effectively getting all the tests without configuration specified. -Disadvantages: it would require manually specifying which tests to run to make the minimum test set bar, which is painful.

Another approach, albeit drastic, would be to switch to CircleCI approach that does build caching on layers and somehow test sections progressively and be able to skip sections that haven't changed. I'm less confident in this approach given our current tree.

[kylepjohnson]

I'm playing with something right now, but just the opposite:

• the bare minimum tests are in the doctests; invoking with pytest --doctest-modules src/cltkv1/ --ignore=tests; this ignores the tests dir in the repo root
• the full suite is run with pytest --doctest-modules src/cltkv1/ tests; this calls the unit tests too

This way, we have the benefit of doctests to help self-document the code. I'd like, as much as possible, to build docs out of the docstrings themselves.

I'm messing with this right now and will post back here what I have done, before signing off. You can give it a once over or even take it up if you wish.

[todd-cook]

I like the way you're thinking! Looking forward to it!

On Mon, Jun 22, 2020 at 9:49 PM Kyle P. Johnson notifications@github.com wrote:

I'm playing with something right now, but just the opposite:

• the bare minimum tests are in the doctests; invoking with pytest --doctest-modules src/cltkv1/ --ignore=tests; this ignores the tests dir in the repo root
• the full suite is run with pytest --doctest-modules src/cltkv1/ tests; this calls the unit tests too

This way, we have the benefit of doctests to help self-document the code. I'd like, as much as possible, to build docs out of the docstrings themselves.

I'm messing with this right now and will post back here what I have done, before signing off. You can give it a once over or even take it up if you wish.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/cltk/cltkv1/issues/66#issuecomment-647905431, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFCOLLDK2A27W55ISFGS73RYAX6XANCNFSM4OEYY2CA .