Some flaky tests - Githubissues

Agorguy commented 1 year ago

Hello,

We tried running your project and discovered that it contains some flaky tests (i.e., tests that nondeterministically pass and fail). We found these tests to fail more frequently when running them on certain machines of ours.

To prevent others from running this project and its tests in machines that may result in flaky tests, we suggest adding information to the README.md file indicating the minimum resource configuration for running the tests of this project as to prevent observation of test flakiness.

If we run this project in a machine with 1cpu and 1gb ram, we observe flaky tests. We found that the tests in this project did not have any flaky tests when we ran it on machines with 2cpu and 2gb ram.

Here is a list of the tests we have identified and their likelihood of failure on a system with less than the recommended 2 CPUs and 2 GB RAM.

com.github.davidmoten.rx2.internal.flowable.FlowableServerSocketTest#testLoad (4 out 10)
com.github.davidmoten.rx2.buffertofile.FlowableOnBackpressureBufferToFileTest#testCancelObservable (1 out 10)

Reproducing


FROM maven:3.5.4-jdk-11

WORKDIR /home/

RUN git clone https://github.com/davidmoten/rxjava2-extras && \
  cd rxjava2-extras && \
  git checkout 3f35966fb5cd3b7a4d4ada38165de3f9fae7c6d6 

WORKDIR /home/rxjava2-extras

RUN mvn install -DskipTests

ENTRYPOINT ["mvn", "test", "-fn"]

Build the image:

$> mkdir tmp

$> cp Dockerfile tmp

$> cd tmp

$> docker build -t rxjava2-extras . # estimated time of build 3m

Running: this configuration likely prevents flakiness (no flakiness in 10 runs)

$> docker run --rm --memory=2g --cpus=4 --memory-swap=-1 rxjava2-extras | tee output.txt
$> grep "Failures:"  output.txt # checking results

checking results

this other configuration –similar to the previous– can’t prevent flaky tests (observation in 10 runs)

$> docker run --rm --memory=1g --cpus=0.5 --memory-swap=-1 rxjava2-extras | tee output2.txt
$> grep "Failures:"  output2.txt # checking results

davidmoten commented 1 year ago

thanks, interesting. Can you post the output of the failing tests?

davidmoten commented 1 year ago

tried those, the supposedly flaky one ran for an hour in a loop without failing so I have no evidence. What's your output?

davidmoten commented 1 year ago

Closing due to lack of further information.

Agorguy commented 1 year ago

Dear @davidmoten, I'm sorry for the delay in responding to you, here are the stack traces of each flaky test at the time of its failure. Please note that these tests fail at a higher rate in environments with a resource constraint, maybe I can create a PR where we add in the README.md a minimum configuration that the user must have to run the tests to avoid presenting flaky?

Agorguy commented 1 year ago

test com.github.davidmoten.rx2.internal.flowable.FlowableServerSocketTest#testLoad

<testcase name="testLoad" classname="com.github.davidmoten.rx2.internal.flowable.FlowableServerSocketTest" time="40.3">
    <failure message="Not completed (latch = 0, values = 999, errors = 1, completions = 0)" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError: Not completed (latch = 0, values = 999, errors = 1, completions = 0)
    at io.reactivex.observers.BaseTestConsumer.fail(BaseTestConsumer.java:189)
    at io.reactivex.observers.BaseTestConsumer.assertComplete(BaseTestConsumer.java:242)
    at com.github.davidmoten.rx2.internal.flowable.FlowableServerSocketTest.testLoad(FlowableServerSocketTest.java:365)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:377)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:284)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:248)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:167)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
Caused by: java.util.concurrent.TimeoutException: The source did not signal an event for 30 seconds and has been terminated.
    at io.reactivex.internal.operators.flowable.FlowableTimeoutTimed$TimeoutSubscriber.onTimeout(FlowableTimeoutTimed.java:139)
    at io.reactivex.internal.operators.flowable.FlowableTimeoutTimed$TimeoutTask.run(FlowableTimeoutTimed.java:170)
    at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
    at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
]]></failure>

Agorguy commented 1 year ago

test com.github.davidmoten.rx2.buffertofile.FlowableOnBackpressureBufferToFileTest#testCancelObservable

<testcase name="testCancelObservable" classname="com.github.davidmoten.rx2.buffertofile.FlowableOnBackpressureBufferToFileTest" time="7.099">
    <failure type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError
    at org.junit.Assert.fail(Assert.java:87)
    at org.junit.Assert.assertTrue(Assert.java:42)
    at org.junit.Assert.assertFalse(Assert.java:65)
    at org.junit.Assert.assertFalse(Assert.java:75)
    at com.github.davidmoten.rx2.buffertofile.FlowableOnBackpressureBufferToFileTest.testCancelObservable(FlowableOnBackpressureBufferToFileTest.java:485)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:377)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:284)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:248)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:167)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
]]></failure>

davidmoten / rxjava2-extras

Some flaky tests #119

Reproducing