ENH: add timeout of 300 (5 minutes) to any test running

yarikoptic commented 6 months ago

We recently started to encounter stalling test runs which lead to hours of stalled operation. Hopefully this would lead to failed test instead of a stall

TODOs:

[x] test if in effect by making timeout too small (e.g. 2 seconds). Will do now

yarikoptic commented 6 months ago

cool -- we do tests failing with

FAILED dandi/cli/tests/test_cmd_ls.py::test_ls_bids_file - Failed: Timeout >3.0s
FAILED dandi/cli/tests/test_cmd_validate.py::test_validate_severity - Failed: Timeout >3.0s
FAILED dandi/tests/test_download.py::test_download_000027[https://dandiarchive.org/dandiset/000027/0.210831.2033] - Failed: Timeout >3.0s
FAILED dandi/tests/test_download.py::test_download_000027[https://dandiarchive.org/dandiset/000027/draft] - Failed: Timeout >3.0s
FAILED dandi/tests/test_metadata.py::test_bids_nwb_metadata_integration - Failed: Timeout >3.0s

Let's hope that it would also ensure to terminate them whenever they run out of time. I think at some point in the past it was not the case with this machinery, but on my trivial test today it seems to be the case:

❯ time python -m pytest -s -v --timeout 10 /tmp/test_sample.py
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.11.9, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /tmp
plugins: anyio-4.3.0, timeout-2.3.1
timeout: 10.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                                                   

../../../../../tmp/test_sample.py::test_foo .
FAILED

===================================================================================== FAILURES =====================================================================================
_____________________________________________________________________________________ test_foo _____________________________________________________________________________________

    @pytest.mark.timeout(1)
    def test_foo():
        for i in range(3):
            print('.')
>           sleep(1)
E           Failed: Timeout >1.0s

/tmp/test_sample.py:8: Failed
============================================================================= short test summary info ==============================================================================
FAILED ../../../../../tmp/test_sample.py::test_foo - Failed: Timeout >1.0s
================================================================================ 1 failed in 1.11s =================================================================================
python -m pytest -s -v --timeout 10 /tmp/test_sample.py  0.51s user 0.07s system 36% cpu 1.580 total

where we see that overall run is less than "planned" 3 seconds of the loop... ok -- removing TEMP commit now

codecov[bot] commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 88.69%. Comparing base (260b4d7) to head (659c237). Report is 2 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #1440 +/- ## ========================================== + Coverage 88.61% 88.69% +0.08% ========================================== Files 77 77 Lines 10563 10563 ========================================== + Hits 9360 9369 +9 + Misses 1203 1194 -9 ``` | [Flag](https://app.codecov.io/gh/dandi/dandi-cli/pull/1440/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/dandi/dandi-cli/pull/1440/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) | `88.69% <ø> (+0.08%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

yarikoptic commented 6 months ago

all good besides the

https://github.com/dandi/dandi-cli/issues/1441 seen also elsewhere. So let's proceed

github-actions[bot] commented 6 months ago

:rocket: PR was released in 0.62.1 :rocket:

dandi / dandi-cli

ENH: add timeout of 300 (5 minutes) to any test running #1440

Codecov Report