SpannerChangeStreamErrorTest.testUnavailableExceptionRetries hard failure (disabled)

damccorm commented 2 years ago

Example failures:


java.lang.AssertionError: 
Expected: a value greater than <1>
     but: <0> was less than <1>
    at
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
    at
org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest.testUnavailableExceptionRetries(SpannerChangeStreamErrorTest.java:166)
    at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:323)
    at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258)
    at
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)

Imported from Jira BEAM-14152. Original Jira may contain additional context. Reported by: lcwik.

kennknowles commented 2 years ago

Confirmed this on my first try: https://scans.gradle.com/s/s3rz2sggpks4m/tests/:sdks:java:io:google-cloud-platform:test/org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest/testUnavailableExceptionRetries?top-execution=1

kennknowles commented 2 years ago

So far 100% failure rate, not flaking.

kennknowles commented 2 years ago

Seems the test is expecting a particular proto class in a mock, so it isn't lining up with expected results

kennknowles commented 2 years ago

@zoercai @thiagotnunes this test is now a hard failure. It looks like it is caused by the way it uses mocking, even though the results of a retry may be fine.

thiagotnunes commented 2 years ago

@nancyxu123

Abacn commented 1 year ago

Looks like the mockSpannerService in SpannerChangeStreamErrorTest actually does not taking effect. It also affects current test suite beam_PreCommit_Java_GCP_IO_Direct where org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest.testAbortedExceptionNotRetriedithDefaultsForStreamSqlRetrySettings costs 5 minutes exactly on success run. The test being success due to real grpc DEADLINE_EXCEEDED.

The same test suite also has high chance of timeout which also possibly due to similar issue.

Reopen the issue for now for mockSpannerService not working properly

apache / beam

SpannerChangeStreamErrorTest.testUnavailableExceptionRetries hard failure (disabled) #21533