t/model_Messages.t fails randomly on Travis CI

[dboehmer]

We've received a failed test report from Travis now for the 2nd time with a weird error.

1. https://travis-ci.org/dboehmer/coocook/builds/635511211 https://travis-ci.org/dboehmer/coocook/builds/637445825 [edit: linked right build) 1 of 2 builds failed at first but passed when I restarted it (invisible now)
2. https://travis-ci.org/dboehmer/coocook/builds/641299279 1 of 2 builds failed but I did only change the README.md.

It seems like Travis CI doesn't show previous builds after restarting them. What a mess.

I suspected this must be a real issue when the test failed and then passed for the first time.

Today's fail report proves there's an issue somewhere.

Ideas:

• something depends on order of hash elements (order is random in Perl for good reasons)

[dboehmer]

I just ran t/model_Messages.t in a loop to verify its result is stable on my machine. The following simple loop passed over 2000 times:

i=1; while prove t/model_Messages.t; do ((i++)); echo $i; done

Edit: Perl version is

This is perl 5, version 30, subversion 0 (v5.30.0) built for x86_64-linux-thread-multi

[dboehmer]

I installed Perl 5.22.4 (thanks to Perlbrew) and found that the test is actually not stable with that Perl version. Sometimes it fails with this error:

$ prove t/model_Messages.t 
t/model_Messages.t .. 1/11 Argument "1/8" isn't numeric in numeric eq (==) at .../coocook/lib/Coocook/Model/Messages.pm line 59.
Argument "1/8" isn't numeric in numeric eq (==) at .../coocook/lib/Coocook/Model/Messages.pm line 59.
Unsupported type at t/model_Messages.t line 26.
# Looks like your test exited with 255 just after 10.
t/model_Messages.t .. Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/11 subtests 

Test Summary Report
-------------------
t/model_Messages.t (Wstat: 65280 Tests: 10 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 11 tests but ran 10.
Files=1, Tests=10,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.06 cusr  0.00 csys =  0.09 CPU)
Result: FAIL

This simple loop

pass=0; for i in {1..1000}; do echo $i; prove t/model_Messages.t && ((pass++)); done; echo "=> $pass passes"

ran 1000 tests with 895 passes. Roughly 10% test failures. Interesting ratio.