search

Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.93k stars 552 forks source link

Time::HiRes t/alarm.t has a flapping test #16251

Open p5pRT opened 6 years ago

p5pRT commented 6 years ago

Migrated from rt.perl.org#132457 (status was 'open')

Searchable as RT132457$

p5pRT commented 6 years ago

From @atoomic

This needs investigation\, unfortunately\, I do not have more details than this at this time. I hope to be able to track it pretty soon.

This is coming from a Travis CI run​: https://travis-ci.org/p5h/perl5demo/jobs/302868505

HEAD is blead + a minor commit which does not seem related to it\, and the test is passing on other macosx builds\, view https://travis-ci.org/p5h/perl5demo/builds/302868502

I assume that the server was running with a high load\, and everything was slow down...

HEAD was here https://github.com/p5h/perl5demo/commit/c4a18419443322ee737a9f018d43a4469dc8fa3f which is blead=4af7d876a8f36e9eda38116fb0b3d5e3fde025f8 + e1f53cba21c7687c50d535433ea1afc8b78259dd

I suggest to identify the test and skip it when ENV{CI} is set\, same as done in cd2a978d2293dbb03a031e76b83063ff03d6a04b ( view PACKAGING for CI option ).

Relevant output from the travis CI log.

____________________________________ # Looks like you failed 1 test of 10. ../dist/Time-HiRes/t/alarm.t ....................................... Dubious\, test returned 1 (wstat 256\, 0x100)

Test Summary Report

../dist/Time-HiRes/t/alarm.t (Wstat​: 256 Tests​: 10 Failed​: 1)   Failed test​: 2   Non-zero exit status​: 1 Files=2627\, Tests=1143283\, 590 wallclock secs (135.12 usr 16.35 sys + 696.97 cusr 123.73 csys = 972.17 CPU) Result​: FAIL make​: *** [test_harness_notty] Error 1

p5pRT commented 6 years ago

From @jkeenan

On Thu\, 16 Nov 2017 20​:14​:32 GMT\, atoomic@​cpan.org wrote​:

This needs investigation\, unfortunately\, I do not have more details than this at this time. I hope to be able to track it pretty soon.

This is coming from a Travis CI run​: https://travis- ci.org/p5h/perl5demo/jobs/302868505

HEAD is blead + a minor commit which does not seem related to it\, and the test is passing on other macosx builds\, view https://travis- ci.org/p5h/perl5demo/builds/302868502

I assume that the server was running with a high load\, and everything was slow down...

HEAD was here https://github.com/p5h/perl5demo/commit/c4a18419443322ee737a9f018d43a4469dc8fa3f which is blead=4af7d876a8f36e9eda38116fb0b3d5e3fde025f8 + e1f53cba21c7687c50d535433ea1afc8b78259dd

I suggest to identify the test and skip it when ENV{CI} is set\, same as done in cd2a978d2293dbb03a031e76b83063ff03d6a04b ( view PACKAGING for CI option ).

Relevant output from the travis CI log.

____________________________________ # Looks like you failed 1 test of 10. ../dist/Time-HiRes/t/alarm.t ....................................... Dubious\, test returned 1 (wstat 256\, 0x100)

Test Summary Report ------------------- ../dist/Time-HiRes/t/alarm.t (Wstat​: 256 Tests​: 10 Failed​: 1) Failed test​: 2 Non-zero exit status​: 1 Files=2627\, Tests=1143283\, 590 wallclock secs (135.12 usr 16.35 sys + 696.97 cusr 123.73 csys = 972.17 CPU) Result​: FAIL make​: *** [test_harness_notty] Error 1

I don't think this warrants any changes in our source code. dist/Time-HiRes/t/alarm.t is a test that not infrequently fails in resource-constrained environments.

For example\, it fails from time to time in the FreeBSD-10.3 VM I maintain on my laptop -- often enough that it's one of four resource-related test failures I cite in a note when I run smoke tests. (See\, e.g.\, http​://perl5.test-smoke.org/report/59803.)

While we may want to have some discussion (on p5p list rather than in RT) of how we handle resource-constrained tests in general\, I don't see a need at this time to change this particular test.

Thank you very much. -- James E Keenan (jkeenan@​cpan.org)

p5pRT commented 6 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 6 years ago

From @xsawyerx

On 11/20/2017 03​:56 PM\, James E Keenan via RT wrote​:

On Thu\, 16 Nov 2017 20​:14​:32 GMT\, atoomic@​cpan.org wrote​:

This needs investigation\, unfortunately\, I do not have more details than this at this time. I hope to be able to track it pretty soon.

This is coming from a Travis CI run​: https://travis- ci.org/p5h/perl5demo/jobs/302868505

HEAD is blead + a minor commit which does not seem related to it\, and the test is passing on other macosx builds\, view https://travis- ci.org/p5h/perl5demo/builds/302868502

I assume that the server was running with a high load\, and everything was slow down...

HEAD was here https://github.com/p5h/perl5demo/commit/c4a18419443322ee737a9f018d43a4469dc8fa3f which is blead=4af7d876a8f36e9eda38116fb0b3d5e3fde025f8 + e1f53cba21c7687c50d535433ea1afc8b78259dd

I suggest to identify the test and skip it when ENV{CI} is set\, same as done in cd2a978d2293dbb03a031e76b83063ff03d6a04b ( view PACKAGING for CI option ).

Relevant output from the travis CI log.

____________________________________ # Looks like you failed 1 test of 10. ../dist/Time-HiRes/t/alarm.t ....................................... Dubious\, test returned 1 (wstat 256\, 0x100)

Test Summary Report ------------------- ../dist/Time-HiRes/t/alarm.t (Wstat​: 256 Tests​: 10 Failed​: 1) Failed test​: 2 Non-zero exit status​: 1 Files=2627\, Tests=1143283\, 590 wallclock secs (135.12 usr 16.35 sys + 696.97 cusr 123.73 csys = 972.17 CPU) Result​: FAIL make​: *** [test_harness_notty] Error 1 I don't think this warrants any changes in our source code. dist/Time-HiRes/t/alarm.t is a test that not infrequently fails in resource-constrained environments.

The problem with this test failing this way (and particularly flapping behavior) is that we do not know whether the function is buggy or whether the system is loaded. Normalizing false positives means ignoring true positives.

I would be happy to detect and prevent these so we could treat a "fail" as "no\, seriously\, this is a bug in the code."

p5pRT commented 6 years ago

From @toddr

On Mon\, 20 Nov 2017 06​:56​:21 -0800\, jkeenan wrote​:

While we may want to have some discussion (on p5p list rather than in RT) of how we handle resource-constrained tests in general\, I don't see a need at this time to change this particular test.

Jim\,

Don't ticket comments relay an email to the p5p list and vice-versa?

+1 to adding the other 3 tests to TODO or whichever on a resource constrained system.

Todd