search

spotify / voyager

🛰️ An approximate nearest-neighbor search library for Python and Java with a focus on ease of use, simplicity, and deployability.
https://spotify.github.io/voyager/
Apache License 2.0
1.26k stars 51 forks source link

Flaky test on CI/CD #65

Closed markkohdev closed 1 month ago

markkohdev commented 4 months ago

The java ConcurrencyTest sporadically fails on CI/CD. We should figure out how to make that test more stable. A short-term workaround is simply re-running the failed build steps, this will usually resolve the issue within 1 retry

Error logs from a flaky run

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.spotify.voyager.jni.ConcurrencyTest
java.lang.RuntimeException: resource deadlock would occur: resource deadlock would occur
    at com.spotify.voyager.jni.Index.addItem(Native Method)
    at com.spotify.voyager.jni.ConcurrencyTest.lambda$itCanAddItemsInParallel$4(ConcurrencyTest.java:180)
    at java.base/java.lang.Thread.run(Thread.java:829)
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.766 sec <<< FAILURE!
itCanAddItemsInParallel(com.spotify.voyager.jni.ConcurrencyTest)  Time elapsed: 1.446 sec  <<< FAILURE!
java.lang.AssertionError: Error in test: resource deadlock would occur: resource deadlock would occur
    at org.junit.Assert.fail(Assert.java:89)
    at com.spotify.voyager.jni.ConcurrencyTest.itCanAddItemsInParallel(ConcurrencyTest.java:200)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Running com.spotify.voyager.jni.IndexTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.354 sec
Running com.spotify.voyager.jni.StringIndexTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.[887](https://github.com/spotify/voyager/actions/runs/8511970940/job/23312645027?pr=63#step:6:888) sec
Running com.spotify.voyager.jni.utils.TinyJsonTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec

Results :

Failed tests:   itCanAddItemsInParallel(com.spotify.voyager.jni.ConcurrencyTest): Error in test: resource deadlock would occur: resource deadlock would occur

Tests run: 26, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:36 min
[INFO] Finished at: 2024-04-01T18:29:21Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project voyager: There are test failures.
Error:  
Error:  Please refer to D:\a\voyager\voyager\java\target\surefire-reports for the individual test results.
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Error: Process completed with exit code 1.
miclegr commented 1 month ago

Closed by #70