search

javapathfinder / jpf-nhandler

GNU General Public License v3.0
5 stars 6 forks source link

jpf-nhandler's tests are failing on `amd` based machines #11

Closed Harsh4902 closed 2 months ago

Harsh4902 commented 3 months ago

35 tests completed, 35 failed

cyrille-artho commented 3 months ago

This is strange, because I have an Intel Mac where the tests pass. Maybe it's the exact JDK version that is slightly different? I have the following:

$ java -version
openjdk version "11.0.23" 2024-04-16
OpenJDK Runtime Environment (build 11.0.23+9)
OpenJDK 64-Bit Server VM (build 11.0.23+9, mixed mode)
Harsh4902 commented 2 months ago

I also have the same configuration, but still it's failing.

openjdk version "11.0.23" 2024-04-16
OpenJDK Runtime Environment (build 11.0.23+9-post-Ubuntu-1ubuntu122.04.1)
OpenJDK 64-Bit Server VM (build 11.0.23+9-post-Ubuntu-1ubuntu122.04.1, mixed mode, sharing)
cyrille-artho commented 2 months ago

Maybe it is an Ubuntu issue then? I have access to one Ubuntu machine that I can use (without docker), and I also get all tests failing there. Ubuntu and Mac OS are normally quite similar when it comes to running Java code, so I'm surprised that Ubuntu causes all tests to fail here. The CI workflow actually also uses Ubuntu, and the last commit had all tests pass.

cyrille-artho commented 2 months ago

On the upside, if you find the reason for one test failure, you probably found them all. I recommend looking into a simple test for this. It's probably just the way the main JPF project (jpf-core) is configured. I am not an expert in Gradle, but I guess the problem lies here (in build.gradle):

    // jpf-core
    implementation files(jpf.Jar.getFullPath())
Harsh4902 commented 2 months ago

@cyrille-artho Yeah I am working on it. I also found one interesting thing, test cases of package converter and converter.specific are working fine. You can test it by using following commands: ./gradlew clean test --tests converter.* ./gradlew clean test --tests converter.specific.*

cyrille-artho commented 2 months ago

You mean they work in isolation? Perhaps something in the other tests changes the global state of the JVM (by setting a static variable). For some reason, I currently can't test JPF at all on the machine I have access to. Gradle does not run.

Harsh4902 commented 2 months ago

You mean they work in isolation? Perhaps something in the other tests changes the global state of the JVM (by setting a static variable). For some reason, I currently can't test JPF at all on the machine I have access to. Gradle does not run.

Yeah, test cases of package converter and converter.specific are running fine when we run it separately by using above-mentioned commands.

cyrille-artho commented 2 months ago

I was able to compile nhandler on Ubuntu 20 (on an older computer), but in that case, all tests fail, regardless of whether some are run in isolation or whether all 35 unit tests are run. However, the order in which Gradle runs tests is not necessarily deterministic, so there may be a test within converter that can interfere with other tests when run early, but perhaps not when most other tests have already concluded. Please keep looking for possible reasons for the test failure. Perhaps jpf-core is not found in my setup, which would explain tests always failing. A test can also "pollute" the environment of other tests by affecting the system-wide configuration through the Java library, so setting static fields (or calling methods that set them) is not even the only way to cause such potential problems.