sado: golden standard broken in sado_test

[richelbilderbeek]

When running sado_test, the golden standard is broken:

Running 50 test cases...
0 1000 -1 -1 1
0.535798 0.484394 0.493132 7.55329e-15 4.44311e-15 9.88593e-15
/usr/include/boost/algorithm/string/find_iterator.hpp:248:36: runtime error: load of value 224, which is not a valid value for type 'bool'
Comparing:
golden  : 0,1000,1,1,1,4.38758e-15,9.88593e-15,7.33114e-15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
measured: 0,1000,-1,-1,1,7.55329e-15,4.44311e-15,9.88593e-15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
sado_test: ../pbdmms/sado_output.cpp:107: void sado::output(const sado::population&, int, const sado::parameters&, sado::results&): Assertion `is_more_or_less_same(golden_values, measured_values)' failed.

In sado_gui however, everything still works as expected

@Lumphie : any idea what would cause this?

[richelbilderbeek]

Tests were disabled at 46ceba6ed3c31bc8426e6dd382c978466fa54985

[Lumphie]

We never changed any individuals right?

[richelbilderbeek]

I assumed so. Looks like I'll have to go back in history and see when/what/when the defect occurred :male_detective:

[richelbilderbeek]

528d46c

[Lumphie]

c9fadf4 This one broke it.

[Lumphie]

dc09033

This one (before it) ran the test and works

[richelbilderbeek]

The diff between correct and incorrect one:

Weird, only the .pro is changed

The run of the correct one:

The test does run, and runs with success

The run of the incorrect one:

The test does run, and runs with failure.

No idea what the problem is, except for seperate_string (sounds rather unlikely)...

[richelbilderbeek]

Locally, I did:

git checkout c9fadf4802df478194f07e1e8c8e60a17807f2f7

Compiled and it worked! At that same computer, in a different folder, I have a further version that does not work. Time for meld :muscle:

[richelbilderbeek]

The 'working' version c9fadf4802df478194f07e1e8c8e60a17807f2f7 does not actually test. Here I proof that the error was already there.

[richelbilderbeek]

So, tests were disabled at 46ceba6ed3c31bc8426e6dd382c978466fa54985, Golden Standard was broken at c9fadf4802df478194f07e1e8c8e60a17807f2f7

Error already present?

• [x] c9fadf4802df478194f07e1e8c8e60a17807f2f7: yes
• [ ] ?
• [ ] 46ceba6ed3c31bc8426e6dd382c978466fa54985: no

Time to use a git bisect

[Lumphie]

But how do you explain that it showed the comparison on travis?

I mean the Result: 1000 1 1 1 4.3...... etc. Golden: 1000 1 1 1 4.3.... etc.

[richelbilderbeek]

richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git checkout 46ceba6
HEAD is now at 46ceba6... Added species class
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git status
HEAD detached at 46ceba6
nothing to commit, working tree clean
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git bisect start
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git bisect good
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git checkout c9fadf4
Previous HEAD position was 46ceba6... Added species class
HEAD is now at c9fadf4... sado_gui needs BoostGraphTutorial
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git bisect bad
Bisecting: 17 revisions left to test after this (roughly 4 steps)
[77dd2aa2f04cc7d95731acd9ffa42754f7a3c277] Progress
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git bisect good
Bisecting: 8 revisions left to test after this (roughly 3 steps)
error: Your local changes to the following files would be overwritten by checkout:
    sado_test.pri
Please commit your changes or stash them before you switch branches.
Aborting
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git checkout sado_test.pri
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git bisect good
Bisecting: 8 revisions left to test after this (roughly 3 steps)
[68d5c98c06d1215a1a53b926a57c0396e798846e] Tests work yay
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git bisect good
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[ed74151fadc1a69c45cef0b35c45a10bf3f49b69] Graph shows species ID and generation, exposes bug better
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git bisect good
Bisecting: 2 revisions left to test after this (roughly 1 step)
[b0056606b891cf408f1ca844b727fa6c5a510f9f] Fixed bug in has_common_ancestor
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git bisect good
Bisecting: 0 revisions left to test after this (roughly 1 step)
[dc090334b594c8d60f46795c2d18e7c4e0a9574b] Test works
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ git bisect good
c9fadf4802df478194f07e1e8c8e60a17807f2f7 is the first bad commit
commit c9fadf4802df478194f07e1e8c8e60a17807f2f7
Author: richelbilderbeek <richel@richelbilderbeek.nl>
Date:   Wed Feb 22 13:23:32 2017 +0100

    sado_gui needs BoostGraphTutorial

:100644 100644 1944f53019f2c04a9104de4d1cce4314eb0ec543 29b7bd9b2519cfa696a2301d9db8739ad4404f98 M  sado_gui.pro
richel@lubuntu:~/GitHubs/sado_works/pbdmms$ 

[richelbilderbeek]

Hmm, c9fadf4802df478194f07e1e8c8e60a17807f2f7 just works. Bisecting again until current version

[richelbilderbeek]

Hypothesis: just clean the build- folder. Incorrect Hypothesis: different makefiles

Failing:

makefiles_fail.zip

OK:

makefiles_ok.zip

Brilliant: the tests that 'pass' do so, because they do not test :confused:

[richelbilderbeek]

What is interesting, is that in the same build log, the test fails and the console run passes

[richelbilderbeek]

It ... works again :confused:

Well, let's label it a Heisenbug and do cool stuff :sunrise:

[richelbilderbeek]

Closing it does not mean I do not experience the bug locally now.

Solution: add a second Golden Standard output file.

[richelbilderbeek]

Ah, it now -of course!- works locally....

All I did:

• renamed get_golden_output to get_golden_output_1
• renamed get_golden_output_1 to get_golden_output

[richelbilderbeek]

Added picture of how it looks like here:

Screenshots/sado_golden.png