search

nanoporetech / scrappie

Scrappie is a technology demonstrator for the Oxford Nanopore Research Algorithms group
Mozilla Public License 2.0
92 stars 27 forks source link

Scrappie unittests fail #24

Closed ccoulombe closed 5 years ago

ccoulombe commented 6 years ago

Hi,

When building scrappie v1.4.1 on linux with Intel/16.0.4 & MKL/11.3.4.258, unit tests fail. Likewise with GCC/5.4.0 & Openblas/0.2.20.

$ make test
Running tests...
Test project /scrappie/build
      Start  1: unittest
 1/19 Test  #1: unittest .........................***Failed    5.84 sec

All others tests passed. I can provide the list of unit tests that fail if that helps.

tmassingham-ont commented 6 years ago

Yes please, and some more details about the platform you are running on would be helpful.

Thank you.

ccoulombe commented 6 years ago

I've replaced all tests that passed with .... The tests were ran with Intel/16.0.4 & MKL/11.3.4.258 on linux:

$ ./scrappie_unittest

     CUnit - A unit testing framework for C - Version 2.1-3
     http://cunit.sourceforge.net/

Suite: Skeleton set of tests for modification
  Test: Skeleton test doing no-op ...passedscrappie_unittest: Failed to open test_matrix.crp to read matrix from.

WARNING - Suite initialization failed for 'Scrappie matrix IO tests'.
...
Suite: Test decoding
  Test: Decoding same as Sloika ...FAILED
    1. scrappie/src/test/test_scrappie_decoding.c:74  - CU_ASSERT_PTR_NOT_NULL_FATAL(post)
  Test: Decoding of original and vectorised posterior same ...FAILED
    1. scrappie/src/test/test_scrappie_decoding.c:36  - CU_ASSERT_PTR_NOT_NULL_FATAL(post)
  Test: Decoding of original and vectorised posterior same with stay penalty ...FAILED
    1. scrappie/src/test/test_scrappie_decoding.c:36  - CU_ASSERT_PTR_NOT_NULL_FATAL(post)
  Test: Decoding of original and vectorised posterior same with skip penalty ...FAILED
    1. scrappie/src/test/test_scrappie_decoding.c:36  - CU_ASSERT_PTR_NOT_NULL_FATAL(post)
...
  Test: Calculation of t-statistic ...scrappie_unittest: Failure at elt 6: 0.000488 0.000000

FAILED
    1. scrappie/src/test/test_scrappie_event_detection.c:118  - CU_ASSERT_TRUE(equality_arrayf(tstat, expt, n, 1e-5))
...
  Test: Event detection is shift-scale invariant ...FAILED
    1. scrappie/src/test/test_scrappie_event_detection.c:175  - CU_ASSERT_EQUAL(ev1.n,ev2.n)
    2. scrappie/src/test/test_scrappie_event_detection.c:178  - CU_ASSERT_DOUBLE_EQUAL(ev1.event[ev].mean * scale + shift,ev2.event[ev].mean,1e-5)
    3. scrappie/src/test/test_scrappie_event_detection.c:178  - CU_ASSERT_DOUBLE_EQUAL(ev1.event[ev].mean * scale + shift,ev2.event[ev].mean,1e-5)
...
  Test: Test mapping -- viterbi ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:137  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- forward ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:148  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- viterbi with path ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:159  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- forwards score exceeds Viterbi ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:184  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- forwards with full band equals forwards ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:224  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- Viterbi with full band equals Viterbi ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:197  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- Viterbi with relaxed band2 equals Viterbi ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:253  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- Viterbi with relaxed band3 equals Viterbi ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:253  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- Viterbi with relaxed band4 equals Viterbi ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:253  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- Viterbi with relaxed band5 equals Viterbi ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:253  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- Viterbi with relaxed band2 equals Viterbi, local pen ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:253  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- Viterbi with relaxed band3 equals Viterbi, local pen ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:253  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- Viterbi with relaxed band4 equals Viterbi, local pen ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:253  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
  Test: Test mapping -- Viterbi with relaxed band5 equals Viterbi, local pen ...FAILED
    1. scrappie/src/test/test_map_to_sequence.c:253  - CU_ASSERT_PTR_NOT_NULL_FATAL(logpost)
...
WARNING - Suite initialization failed for 'Test manipulating signal'.
...

Run Summary:    Type  Total    Ran Passed Failed Inactive
              suites     11      9    n/a      2        0
               tests     71     64     44     20        0
             asserts    437    437    415     22      n/a

Elapsed time =    0.017 seconds
tmassingham-ont commented 6 years ago

Thanks. Most of these errors seem to correspond to a failure to read test data. Do you get the same results when you run the scrappie_unittest program from within the src/test folder?

ccoulombe commented 6 years ago

Less tests have failed.

$ pwd
scrappie/src/test
$ ./scrappie_unittest 

     CUnit - A unit testing framework for C - Version 2.1-3
     http://cunit.sourceforge.net/

...
Suite: Test decoding
  Test: Decoding same as Sloika ...passed
  Test: Decoding of original and vectorised posterior same ...passed
  Test: Decoding of original and vectorised posterior same with stay penalty ...FAILED
    1. scrappie/src/test/test_scrappie_decoding.c:48  - CU_ASSERT_DOUBLE_EQUAL(score_original,score_vectorised,1e-5)
  Test: Decoding of original and vectorised posterior same with skip penalty ...passed
...
Suite: Tests for event detection ported from Dragonet
  Test: Cumulative sum and sums ...passed
  Test: Calculation of t-statistic ...scrappie_unittest: Failure at elt 6: 0.000488 0.000000

FAILED
    1. scrappie/src/test/test_scrappie_event_detection.c:118  - CU_ASSERT_TRUE(equality_arrayf(tstat, expt, n, 1e-5))
...
  Test: Event detection is shift-scale invariant ...FAILED
    1. scrappie/src/test/test_scrappie_event_detection.c:175  - CU_ASSERT_EQUAL(ev1.n,ev2.n)
    2. scrappie/src/test/test_scrappie_event_detection.c:178  - CU_ASSERT_DOUBLE_EQUAL(ev1.event[ev].mean * scale + shift,ev2.event[ev].mean,1e-5)
    3. scrappie/src/test/test_scrappie_event_detection.c:178  - CU_ASSERT_DOUBLE_EQUAL(ev1.event[ev].mean * scale + shift,ev2.event[ev].mean,1e-5)
Suite: Test map to sequence code, including banded routines
  Test: Test assumptions about bounds -- not null ...scrappie_unittest: One or more bounds are NULL

scrappie_unittest: One or more bounds are NULL

scrappie_unittest: One or more bounds are NULL

passed
  Test: Test assumptions about bounds -- simple pass ...passed
  Test: Test assumptions about bounds -- includes zero ...scrappie_unittest: First bound must include 0 (got 1)

scrappie_unittest: Blocks 3 and 4 don't overlap [3 , 3) -> [4 , 6)

passed
  Test: Test assumptions about bounds -- includes last position ...scrappie_unittest: Last bound must equal seqlen 6 (got 4)

scrappie_unittest: Blocks 0 and 1 don't overlap [0 , 0) -> [1 , 1)

scrappie_unittest: Blocks 1 and 2 don't overlap [1 , 1) -> [2 , 2)

scrappie_unittest: Blocks 2 and 3 don't overlap [2 , 2) -> [3 , 3)

...

Run Summary:    Type  Total    Ran Passed Failed Inactive
              suites     11     11    n/a      0        0
               tests     71     71     68      3        0
             asserts    508    508    503      5      n/a
tmassingham-ont commented 6 years ago

Great. These remaining test look like tolerance issues on the tests. I'll try to replicate: what changes did you make to use ICC / MKL please?

ccoulombe commented 6 years ago

To reproduce, you can clone this branch: From the HEAD of intel_mkl branch: git diff HEAD^

Following the compiling instructions in the README: cmake -DHDF5_ROOT=/path/to/HDF5_dir -DCMAKE_C_COMPILER=/path/to/icc -DCMAKE_C_FLAGS=-qopenmp ..

So yes, tests were ran from the build directory.

tmassingham-ont commented 6 years ago

The tests work for me using gcc 5.4.0 on Ubuntu 16.04 using both OpenBLAS 0.2.18 and MKL 2019.0.117 (the latest version I could get hold of), which leaves the Intel compiler as the probable cause of the difference. I'm trying to get access to a licenced copy of the compiler at the moment but, in the meantime, do you still have issues with GCC + MKL when running scrappie_unittest in the src/test directory (or just running make test at the top level)?

ccoulombe commented 5 years ago

GCC works, I don't get why it didn't the first time. Even from the build directory. Intel does not pass tests from the build directory, nor when copied in the test directory as there's tolerance issues as pointed above.