Open AlbertoEAF opened 2 years ago
Apparently the issue is somewhere in these tests related to LGBM feature contributions, from the LightGBMBinaryClassificationModelTrainerTest
test class:
testFeatureContributionsTargetBeginning
testFeatureContributionsTargetMiddle
testFeatureContributionsTargetEnd
Disabling them lets tests run all the time with mvn test
. Can you review if we're doing something wrong in the Java provider @mlobofeedzai ?
Otherwise, I'm afraid it's a bug at the LGBM level related with fast prediction handles, and having several to our booster: https://github.com/microsoft/LightGBM/issues/4980#issuecomment-1039038250, which would be much trickier to fix.
As of #121 LightGBM tests can still fail after the first run with a serious C++ error from the LightGBM side:
After that the following messages will show up:
How to reproduce
Tests will always run with:
However, they will always fail (at least after the first run) with:
Conclusion
surefire settings: How to have
mvn test
work all the timeBy simply changing maven-surefire-plugin's forking settings (see [1]) it's possible to have
mvn test
work all the time too.This is the observed behaviour regarding
mvn test
:Update: All fork settings can still fail after all. The failure odds change though. The only command that works all the time is still
mvn clean test
. This means the approaches described below don't fix the issue.Analysis
Given that disabling forks altogether vs running the tests in a forked process (default) causes the issues, the problem seems caused by Java's test stack and not by the LightGBM C++ side (which always works with
mvn clean test
anyway).Note: Upgrading to the latest versions of jacoco 0.8.6 and maven-surefire-plugin to 3.0.0-M6 didn't fix the issue either.
Possible solutions
Now, we can disable forks altogether, instead of forking the process before starting the tests once (default, which is failing); or we can set tests not to reuse forks. The latter option is slower as it forks the process for every test class.
Are there any downsides to doing these approaches? And which of those approaches is best?
References
1. maven-surefire-plugin forking docs