search

openmm / NNPOps

High-performance operations for neural network potentials
Other
81 stars 17 forks source link

Many tests are being skipped by the CI #98

Closed RaulPPelaez closed 1 year ago

RaulPPelaez commented 1 year ago

The CI runs this to test:

ctest --verbose --exclude-regex TestCuda

Running this on my machine results in 11 tests. But the CI only runs 3. I cannot find out why.

mikemhenry commented 1 year ago

It looks like we run 5 on the GPU CI: https://github.com/openmm/NNPOps/actions/runs/5063145679/jobs/9089458222#step:9:75 which just calls ctest --verbose

RaulPPelaez commented 1 year ago

I think I know whats going on: https://github.com/openmm/NNPOps/blob/054d487d9fec8f98a111bc30ec6d3ec1ce423356/CMakeLists.txt#L60-L77

This assumes that the custom target runs before the loop in line 74. Since this is not the case, the first time one runs cmake there is no file under CMAKE_BINARY_DIR/test and thus the tests are skipped. I just tried and confirmed this is the case, run cmake once and you get 3 tests, run twice and you get 11.

RaulPPelaez commented 1 year ago

I just want to point out that I gave the above snippet to gpt4 without any context and the thing answered this:

There seems to be an issue here, though. The add_test command is run at configure time, but the files that you want to glob won't exist until build time, after the custom command is run. Therefore, file(GLOB_RECURSE PYTHON_TEST_PATHS ${CMAKE_BINARY_DIR}/test/Test*.py) will return an empty list.

To fix this issue, you could move the test discovery and registration into a separate script that you call using add_test, or consider configuring the tests at build time. This is a known limitation of CMake's test discovery.

Dang... I opened #104 to work on this.