t/integration/reload_syntax_error.t fails since 1.000095 on slow machines

[ppisar]

Since 1.000095 I experience a repetitive test failure in t/integration/reload_syntax_error.t:

./t/integration/replay.t ................... ok
    # Failed test 'Exit Value Check'
    # at ./t/integration/reload_syntax_error.t line 83.
    # +-----+----+-------+
    # | GOT | OP | CHECK |
    # +-----+----+-------+
    # | 256 | eq | 0     |
    # +-----+----+-------+
    # Command = /usr/bin/perl /usr/bin/yath -D/usr/share/perl5/vendor_perl run ./t/integration/reload_syntax_error.tx
    # Exit = 256
    # ==== Output ====
    # (INTERNAL)     19086 yath-nested-runner-A - Runner detected a change in one or more preloaded modules...
    # (INTERNAL)     19086 yath-nested-runner-A - blacklisting changed files and reloading stage...
    # (INTERNAL)     Global symbol "$bob" requires explicit package name (did you forget to declare "my $bob"?) at /tmp/rZx4DpKaWI/Preload/Flux.pm line 7.
    # (INTERNAL)     Compilation failed in require at (eval 168) line 1.
    # ( FAILED )  job  1    t/integration/reload_syntax_error.tx
    # < REASON >  job  1    Test script returned error (Signal: 1)
    # < REASON >  job  1    No plan was declared, and no assertions were made.
    # 
    # The following jobs failed:
    # +------------------------------------+------------------------------------+
    # | Job ID                             | Test File                          |
    # +------------------------------------+------------------------------------+
    # | 854D2D3E-8017-11EC-995A-60A139B4B3 | t/integration/reload_syntax_error. |
    # | 6F                                 | tx                                 |
    # +------------------------------------+------------------------------------+
    # 
    #                                 Yath Result Summary
    # -----------------------------------------------------------------------------------
    #      Fail Count: 1
    #      File Count: 1
    # Assertion Count: 0
    #       Wall Time: 1.63 seconds
    #        CPU Time: 0.89 seconds (usr: 0.29s | sys: 0.12s | cusr: 0.29s | csys: 0.19s)
    #       CPU Usage: 54%
    #     -->  Result: FAILED  <--
    # [0m
    # 
    # ========

# Failed test 'yath run ./t/integr[...]x_error.tx'
# at ./t/integration/reload_syntax_error.t line 83.
./t/integration/resource.t ................. ok
    # Failed test 'Exit Value Check'
    # at ./t/integration/reload_syntax_error.t line 91.
    # +-----+----+-------+
    # | GOT | OP | CHECK |
    # +-----+----+-------+
    # | 256 | eq | 0     |
    # +-----+----+-------+
    # Command = /usr/bin/perl /usr/bin/yath -D/usr/share/perl5/vendor_perl run ./t/integration/reload_syntax_error.tx :: fixed
    # Exit = 256
    # ==== Output ====
    # (INTERNAL)     19086 yath-nested-runner-A - Runner detected a change in one or more preloaded modules...
    # (INTERNAL)     19086 yath-nested-runner-A - blacklisting changed files and reloading stage...
    # (INTERNAL)     19103 yath-nested-runner-A - Runner detected a change in one or more preloaded modules...
    # (INTERNAL)     19103 yath-nested-runner-A - blacklisting changed files and reloading stage...
    # (INTERNAL)     Global symbol "$bob" requires explicit package name (did you forget to declare "my $bob"?) at /tmp/rZx4DpKaWI/Preload/Flux.pm line 7.
    # (INTERNAL)     Compilation failed in require at (eval 168) line 1.
    # ( FAILED )  job  1    t/integration/reload_syntax_error.tx
    # < REASON >  job  1    Test script returned error (Signal: 1)
    # < REASON >  job  1    No plan was declared, and no assertions were made.
    # 
    # The following jobs failed:
    # +------------------------------------+------------------------------------+
    # | Job ID                             | Test File                          |
    # +------------------------------------+------------------------------------+
    # | 878AB4E0-8017-11EC-A029-6EA139B4B3 | t/integration/reload_syntax_error. |
    # | 6F                                 | tx                                 |
    # +------------------------------------+------------------------------------+
    # 
    #                                 Yath Result Summary
    # -----------------------------------------------------------------------------------
    #      Fail Count: 1
    #      File Count: 1
    # Assertion Count: 0
    #       Wall Time: 1.64 seconds
    #        CPU Time: 0.90 seconds (usr: 0.29s | sys: 0.13s | cusr: 0.31s | csys: 0.17s)
    #       CPU Usage: 54%
    #     -->  Result: FAILED  <--
    # [0m
    # 
    # ========

# Failed test 'yath run ./t/integr[...]x_error.tx :: fixed'
# at ./t/integration/reload_syntax_error.t line 91.
# Seeded srand with seed '20220128' from local date.
./t/integration/reload_syntax_error.t ...... 
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/5 subtests

I observe this failure in Fedora CI (complete log at https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/106391/testReport/(root)/tests/_tests_upstream_tests/ will disappear soon), but I cannot reproduce it locally. I also don't have a shell access to the CI machine to debug it more. All other tests pass. The test is about reloading a preloaded module which is an invalid Perl code (an undeclared lexical variable).

I noticed the test contains "sleep 2". This is a typical place where a code becomes sensitive to a real time. I removed the sleep, but still could not reproduce it locally. Interesting fact is that the test is quite slow even without that sleep. Maybe there is another sleep in the library.

Therefore I increased the sleep 5 times and then the test passed in the CI system. Any idea how to fix it properly?

[ppisar]

The longer sleep did not help. It passed for the first time probably by a chance.

[exodist]

I am going to make this an AUTHOR_TESTING only test. It is useful for me when maintaining features and catching bugs, but the nature of the test makes it suffer from timing problems (that will not effect things outside of tests). I will still run it for releases, but no need to block installation on it.

[ppisar]

Thanks. That would resolve the problem for me. However, wouldn't be better to move the test file into ./xt directory?