apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.88k stars 4.26k forks source link

[Disabled]: beam_PostCommit_Java_DataflowV2 SpannerReadIT multiple test failing #26616

Open Abacn opened 1 year ago

Abacn commented 1 year ago

What happened?

e.g. https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_DataflowV2/lastCompletedBuild/

Following tests are permared:

 org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.testReadWithTimeoutError
 org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.testReadFailsBadTable
 org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.testReadFailsBadSession
 org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.testQueryWithTimeoutError

error message: These are all error-reproducing tests failing with expected error not seen. Likely due to recent Spanner service rollout.

java.lang.AssertionError: 
Expected: (stack trace contains string 'SpannerException' and stack trace contains string 'DEADLINE_EXCEEDED')
     but: stack trace contains string 'SpannerException' was <java.lang.RuntimeException: This message was redacted. Please
search the job logs in Cloud Logging for the entry with unique prefix: OutputMessage[5844617984962013488] for work item
1120611541754862560This message was redacted. Please search the job logs in Cloud Logging for the entry with unique
prefix: OutputMessage[9250666589450804496] for work item 5143587066297128187This message was redacted. Please
search the job logs in Cloud Logging for the entry with unique prefix: OutputMessage[5844617984962011739] for work item
6016024597067538899This message was redacted. Please search the job logs in Cloud Logging for the entry with unique
prefix: OutputMessage[9250666589450803181] for work item 8190027929335299829Workflow failed. Causes: S13:read
db/Execute query/Reshuffle/Reshuffle/GroupByKey/Read+read db/Execute query/Reshuffle/Reshuffle/GroupByKey/GroupByWindow+read db/Execute 
...

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

Abacn commented 1 year ago

Still one test failing:

21:28:54 org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT > testReadWithDataBoost FAILED
21:28:54     java.lang.RuntimeException at SpannerReadIT.java:214

And the test suite timing out consistently since May 15th

Update: the test suite timing out also due to this test. From Dataflow UI, this test stuck indefinitely:

spannerreadit0testreadwithdataboost-jenkins-0517114152-fb3ba8ca

Elapsed time
1 hr 19 min