Expected test results problem

[akesandgren]

Hi!

It would be good to have some info on which compiler, version, and flags (in case fobos settings change in the future) the results in TestCases_Results_expected/ are created with. And exactly which version/commit of the code(s) where used to create them. Preferably in a README in the tests folder.

The code seems vulnerable to compiler versions/flags and produces small to medium numerical differences according to tests.py

And the same info should be provided for libaed2 since that also affects the results as far as I can see.

[f-baerenbold]

Hello, Thanks for your suggestion. I will do this for future releases.

In general, the expected testcase-results folder is not up-to-date on the master, only for releases.

[akesandgren]

We're using the release but still can't get the same results with the used GCC version, or any other one I've tried either.

[f-baerenbold]

Which release? windows or linux?

[akesandgren]

Linux

[f-baerenbold]

and which release?

[akesandgren]

3.01

[f-baerenbold]

Ok, thanks. I'll have a look at this soon.

Are the differences significant for temperature? I.e. do you get differences > 0.1 °C or is it more like numerical differences?

[akesandgren]

test.py gives me lots of errors, like

NN_out.dat @ 35091.000: 1.00e-10, 4.29e-03
ERROR: NN_out.dat @ 35091.000: max error = 1.00e-10, max rel error = 4.29e-03
NN_out.dat @ 35325.333: 8.70e-15, 1.68e-03
NN_out.dat @ 35385.333: 3.48e-08, 4.79e-03
ERROR: NN_out.dat @ 35385.333: max error = 3.48e-08, max rel error = 4.79e-03
NN_out.dat @ 35419.000: 2.90e-09, 3.82e-03
ERROR: NN_out.dat @ 35419.000: max error = 2.90e-09, max rel error = 3.82e-03
NN_out.dat @ 35419.333: 1.16e-07, 6.39e-02
ERROR: NN_out.dat @ 35419.333: max error = 1.16e-07, max rel error = 6.39e-02
...
P_out.dat @ 35419.000: 1.35e-10, 3.50e-03
ERROR: P_out.dat @ 35419.000: max error = 1.35e-10, max rel error = 3.50e-03
P_out.dat @ 35419.333: 1.03e-09, 4.81e-02
ERROR: P_out.dat @ 35419.333: max error = 1.03e-09, max rel error = 4.81e-02
P_out.dat @ 35419.667: 2.41e-10, 6.19e-02
ERROR: P_out.dat @ 35419.667: max error = 2.41e-10, max rel error = 6.19e-02
...
num_out.dat @ 35003.333: 1.40e-07, 1.06e-03
ERROR: num_out.dat @ 35003.333: max error = 1.40e-07, max rel error = 1.06e-03
num_out.dat @ 35003.667: 1.94e-07, 2.56e-03
ERROR: num_out.dat @ 35003.667: max error = 1.94e-07, max rel error = 2.56e-03
num_out.dat @ 35004.333: 7.70e-08, 1.04e-03
ERROR: num_out.dat @ 35004.333: max error = 7.70e-08, max rel error = 1.04e-03
...
num_out.dat @ 35003.333: 1.40e-07, 1.06e-03
ERROR: num_out.dat @ 35003.333: max error = 1.40e-07, max rel error = 1.06e-03
num_out.dat @ 35003.667: 1.94e-07, 2.56e-03
ERROR: num_out.dat @ 35003.667: max error = 1.94e-07, max rel error = 2.56e-03
num_out.dat @ 35004.333: 7.70e-08, 1.04e-03
ERROR: num_out.dat @ 35004.333: max error = 7.70e-08, max rel error = 1.04e-03
...
ERROR: Output file TestCases_Results/CAR_ch4_bub_out.dat missing
...
ERROR: NN_out.dat: expected
35003.0000,NaN,2.2256E-008,1.0312E-007,2.1256E-007,3.7021E-007,5.4110E-007,6.4397E-007,6.0926E-007,4.5692E-007,2.7767E-007,1.4215E-007,6.3749E-008,2.5611E-008,9.5590E-009,2.8751E-009,3.0634E-010,-3.0088E-010,-3.2173E-010,8.2725E-010,2.2161E-009,3.7564E-009,5.5627E-009,8.2984E-009,1.7989E-008,3.0694E-008,5.9565E-008,1.3707E-007,3.5242E-007,7.0954E-007,8.7106E-007,9.1175E-007,9.1407E-007,9.0409E-007,9.8240E-007,9.7394E-007,9.9004E-007,1.1334E-006,1.9285E-006,3.1297E-006,3.3447E-006,3.4788E-006,3.6164E-006,3.7539E-006,3.8914E-006,4.0289E-006,4.1751E-006,4.2054E-006,1.8280E-006,2.5074E-007,6.0486E-008,2.9484E-008,1.7964E-008,1.1689E-008,8.1407E-009,5.7024E-009,3.9689E-009,2.8388E-009,2.0523E-009,1.4961E-009,1.1011E-009,8.2303E-010,6.3679E-010,5.4107E-010,1.7889E-010,-1.7493E-011,-1.1716E-010,-1.6688E-010,-1.8063E-010,-1.3079E-010,-6.2761E-011,1.0037E-010,8.6605E-010,7.5955E-009,5.7045E-010,-1.4373E-010,-3.5141E-010,-4.6201E-010,-4.7662E-010,-4.2384E-010,-3.8474E-010,-3.4548E-010,-3.7096E-010,-2.6700E-010,-2.5223E-010,-2.1731E-010,-1.7752E-010,-1.3731E-010,-9.4747E-011,-5.1110E-011,-6.7556E-012,4.4213E-011,1.2225E-010,3.2241E-010,1.4270E-010,8.7009E-011,6.7005E-011,7.0133E-011,1.0084E-010,6.5304E-011,5.7715E-011,6.9369E-011,1.0383E-010,1.8293E-010,1.8797E-010,2.5673E-010,4.4313E-010,1.0468E-009,4.5919E-009,1.1269E-009,1.1257E-009,2.2095E-009,5.8059E-009,1.2832E-008,1.1413E-009,-9.4805E-010,-1.2535E-009,-1.5195E-009,-1.6291E-009,-1.2007E-009,-8.5234E-010,-6.0266E-010,1.4908E-009,9.0929E-008,4.3510E-006,1.0913E-005,1.2279E-005,1.2698E-005,1.2764E-005,1.3460E-005,1.3308E-005,1.3669E-005,1.7942E-005,1.2025E-004,4.4472E-005,5.2110E-005,1.9321E-006,1.0401E-006,1.2529E-006

but was
35003.0000,NaN,2.2256E-008,1.0312E-007,2.1256E-007,3.7021E-007,5.4110E-007,6.4397E-007,6.0926E-007,4.5692E-007,2.7767E-007,1.4215E-007,6.3749E-008,2.5611E-008,9.5590E-009,2.8751E-009,3.0634E-010,-3.0088E-010,-3.2173E-010,8.2725E-010,2.2161E-009,3.7564E-009,5.5627E-009,8.2984E-009,1.7989E-008,3.0694E-008,5.9565E-008,1.3707E-007,3.5242E-007,7.0954E-007,8.7106E-007,9.1175E-007,9.1407E-007,9.0409E-007,9.8240E-007,9.7394E-007,9.9004E-007,1.1334E-006,1.9285E-006,3.1297E-006,3.3447E-006,3.4788E-006,3.6164E-006,3.7539E-006,3.8914E-006,4.0289E-006,4.1751E-006,4.2054E-006,1.8280E-006,2.5074E-007,6.0486E-008,2.9484E-008,1.7964E-008,1.1689E-008,8.1407E-009,5.7024E-009,3.9689E-009,2.8388E-009,2.0523E-009,1.4961E-009,1.1011E-009,8.2303E-010,6.3679E-010,5.4107E-010,1.7889E-010,-1.7493E-011,-1.1716E-010,-1.6688E-010,-1.8063E-010,-1.3079E-010,-6.2761E-011,1.0037E-010,8.6605E-010,7.5955E-009,5.7045E-010,-1.4373E-010,-3.5141E-010,-4.6202E-010,-4.7662E-010,-4.2384E-010,-3.8475E-010,-3.4550E-010,-3.7096E-010,-2.6700E-010,-2.5221E-010,-2.1734E-010,-1.7752E-010,-1.3731E-010,-9.4747E-011,-5.1110E-011,-6.7556E-012,4.4213E-011,1.2225E-010,3.2241E-010,1.4270E-010,8.7009E-011,6.7005E-011,7.0133E-011,1.0084E-010,6.5304E-011,5.7715E-011,6.9369E-011,1.0383E-010,1.8293E-010,1.8797E-010,2.5673E-010,4.4313E-010,1.0468E-009,4.5919E-009,1.1269E-009,1.1257E-009,2.2095E-009,5.8059E-009,1.2832E-008,1.1413E-009,-9.4805E-010,-1.2535E-009,-1.5195E-009,-1.6291E-009,-1.2007E-009,-8.5234E-010,-6.0268E-010,1.4908E-009,9.0929E-008,4.3510E-006,1.0913E-005,1.2279E-005,1.2698E-005,1.2764E-005,1.3460E-005,1.3308E-005,1.3669E-005,1.7942E-005,1.2025E-004,4.4472E-005,5.2110E-005,1.9321E-006,1.0401E-006,1.2529E-006

Not sure if that's severe enough to care about or not. Since it does say ERROR in upper case it hints at it being a bad problem.

[f-baerenbold]

I checked the history of the expected results and it seems that they have not been updated prior to the release of version 3.01. So it is to be expected that there are some minor differences depending on the changes.. Sorry for that. I'll try to make sure that the testcase results are always updated for each release and I'll integrate some information about compiler version and flags.

[akesandgren]

Ok, so we can currently ignore the result discrepancies above?

[f-baerenbold]

Yes, the changes are most probably due to small updates of the code since the last update of the expected results.