testing/runUnitTests may fail due to tolerances

[svigerske]

Issue created by migration from Trac.

Original creator: buchner

Original creation time: 2014-07-29 13:06:15

Assignee: ferreau

Version: 3.0.0

For me 4 of the tests fail.

Running unit test ../bin/test_matrices ... failed! Running unit test ../bin/test_matrices2 ... passed! Running unit test ../bin/test_matrices3 ... passed! Running unit test ../bin/test_indexlist ... passed! Running unit test ../bin/test_example1 ... passed! Running unit test ../bin/test_example1a ... failed! Running unit test ../bin/test_example1b ... passed! Running unit test ../bin/test_example2 ... failed! Running unit test ../bin/test_example4 ... passed! Running unit test ../bin/test_example5 ... passed! Running unit test ../bin/test_exampleLP ... passed! Running unit test ../bin/test_qrecipe ... failed! Running unit test ../bin/test_example6 ... passed! Running unit test ../bin/test_example7 ... passed! Running unit test ../bin/test_sebastien1 ... passed! Running unit test ../bin/test_vanBarelsUnboundedQP ... passed! Running unit test ../bin/test_janick1 ... passed! Running unit test ../bin/test_janick2 ... passed! Running unit test ../bin/test_constraintProduct1 ... problem data missing! Running unit test ../bin/test_constraintProduct2 ... problem data missing! Running unit test ../bin/test_guessedWS1 ... passed! Running unit test ../bin/test_gradientShift ... passed! Running unit test ../bin/test_runAllOqpExamples ... problem data missing! Running unit test ../bin/test_bench Odd... problem data missing! Running unit test ../bin/test_bench Ods... problem data missing! Running unit test ../bin/test_bench Omd... problem data missing! 4 test(s) failed!

[svigerske]

Comment by ferreau created at 2014-07-29 16:00:03

Alex, can you specify your operating system details? Moreover, can you check why the tests are failing on your machine? Are the tolerances violated by tiny amounts or are results completely wrong?

[svigerske]

Comment by ferreau created at 2014-07-29 16:00:03

Changing status from new to assigned.

[svigerske]

Comment by ferreau created at 2014-07-29 16:02:13

This is the output on Ubuntu 12.04, 32-bit, compiled with default settings (i.e. using BLAS/LAPACKreplacements) using gcc 4.6.3:

Running unit test ../bin/test_matrices ... passed! Running unit test ../bin/test_matrices2 ... passed! Running unit test ../bin/test_matrices3 ... passed! Running unit test ../bin/test_indexlist ... passed! Running unit test ../bin/test_example1 ... passed! Running unit test ../bin/test_example1a ... passed! Running unit test ../bin/test_example1b ... passed! Running unit test ../bin/test_example2 ... passed! Running unit test ../bin/test_example4 ... passed! Running unit test ../bin/test_example5 ... passed! Running unit test ../bin/test_exampleLP ... passed! Running unit test ../bin/test_qrecipe ... passed! Running unit test ../bin/test_example6 ... passed! Running unit test ../bin/test_example7 ... passed! Running unit test ../bin/test_sebastien1 ... passed! Running unit test ../bin/test_vanBarelsUnboundedQP ... passed! Running unit test ../bin/test_janick1 ... passed! Running unit test ../bin/test_janick2 ... passed! Running unit test ../bin/test_constraintProduct1 ... problem data missing! Running unit test ../bin/test_constraintProduct2 ... problem data missing! Running unit test ../bin/test_guessedWS1 ... passed! Running unit test ../bin/test_gradientShift ... passed! Running unit test ../bin/test_runAllOqpExamples ... problem data missing! Running unit test ../bin/test_bench Odd... problem data missing! Running unit test ../bin/test_bench Ods... problem data missing! Running unit test ../bin/test_bench Omd... problem data missing!

[svigerske]

Comment by buchner created at 2014-07-30 08:48:11

My system is Ubuntu 14.04, 64 bit, gcc 4.8.2.

After downloading the missing tests, for me 7 tests fail.

Right now I just checked the first three failing tests. There the problems are the tolerances.

../bin/test_matrices ... failed!: errT = 1,2434497875801753e-14 > 1e-14 ../bin/test_example1a ... failed! stat = 1.776357e-15 > 1e-15 ../bin/test_example2 ... failed! maxKKTviolation: 1.776357e-15 > 1e-15

Running unit test ../bin/test_matrices ... failed! Running unit test ../bin/test_matrices2 ... passed! Running unit test ../bin/test_matrices3 ... passed! Running unit test ../bin/test_indexlist ... passed! Running unit test ../bin/test_example1 ... passed! Running unit test ../bin/test_example1a ... failed! Running unit test ../bin/test_example1b ... passed! Running unit test ../bin/test_example2 ... failed! Running unit test ../bin/test_example4 ... passed! Running unit test ../bin/test_example5 ... passed! Running unit test ../bin/test_exampleLP ... passed! Running unit test ../bin/test_qrecipe ... failed! Running unit test ../bin/test_example6 ... passed! Running unit test ../bin/test_example7 ... passed! Running unit test ../bin/test_sebastien1 ... passed! Running unit test ../bin/test_vanBarelsUnboundedQP ... passed! Running unit test ../bin/test_janick1 ... passed! Running unit test ../bin/test_janick2 ... passed! Running unit test ../bin/test_constraintProduct1 ... passed! Running unit test ../bin/test_constraintProduct2 ... passed! Running unit test ../bin/test_guessedWS1 ... passed! Running unit test ../bin/test_gradientShift ... passed! Running unit test ../bin/test_runAllOqpExamples ... failed! Running unit test ../bin/test_bench Odd... failed! Running unit test ../bin/test_bench Ods... failed! Running unit test ../bin/test_bench Omd... passed! 7 test(s) failed!

[svigerske]

Comment by buchner created at 2014-07-30 09:10:39

For test_qrecipe it's also about the tolerances.

And for the other three tests there are problems with opening the files or finding the required data.

[svigerske]

Comment by ferreau created at 2014-07-30 12:09:11

Yes, we should add some provision to make the tolerances somewhat adjustable. Also compiling with -O3 or -g makes a difference.

Can you try to spot the exact problem with reading problem data in the other tests? Moreover, it looks strange that the last one passes but the two other test_bench run do not as they all work on the same problems/data, just with different options.

[svigerske]

Comment by buchner created at 2014-08-01 08:54:00

In test_runAllOqpExamples there were wrong paths to the files. I fixed them in rev 32.

Now it still fails, but because of tolerances in the crane example.

I will give further feedback for test_bench Odd and Ods soon.

[svigerske]

Comment by buchner created at 2014-08-01 10:18:55

Here's the feedback for test_bench:

Odd and Ods failed for not finding the test problems. I fixed that problem. Now they pass all 44 tests, but the tests in general fail because of expectedAvgStationarity
expectedAvgavgFeasibility expectedAvgFeasibility.

Omd is really weird. It passed 43 of 44 tests but is only expected to pass expectedNumSolvedProblems = 42 (?!) and so passes in general. Is this on purpose?

[svigerske]

Comment by ferreau created at 2014-08-04 13:40:57

Regarding test_bench -Omd: On my machine (cygwin/g++), this test "solves" all 44 problems, but two of them are marked as "failed" as solution does not satisfy required accuracy. So, it is not surprising that the number of solved problems may slightly change on different platforms.

As there seems to be no real bug left, I am changing type to "code cleanup": we need to implement some provision to adapt required accuracy to the actual HW/SW setup.

[svigerske]

Comment by ferreau created at 2014-08-04 13:40:57

Changing type from test request to code cleanup.

[svigerske]

Comment by ferreau created at 2014-12-17 13:43:46

I have added a constant TEST_TOL_FACTOR that may be set at compile time (default is 1). All test tolerances are multiplied by this factor before making the comparison.

I am going to close this ticket. Yet suggestions for better solutions to this issue are still welcome.

[svigerske]

Comment by ferreau created at 2014-12-17 13:43:46

Resolution: fixed