apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.69k stars 4.2k forks source link

The PostCommit Java ValidatesRunner Samza job is flaky #31725

Closed github-actions[bot] closed 6 days ago

github-actions[bot] commented 2 weeks ago

The PostCommit Java ValidatesRunner Samza is failing over 50% of the time. Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Samza.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs. See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=3&var-Workflow=PostCommit%20Java%20ValidatesRunner%20Samza

Abacn commented 1 week ago
testWriteUnboundedWithCustomBatchParameters (org.apache.beam.sdk.io.TextIOWriteTest) failed
runners/samza/build/test-results/validatesRunner/TEST-org.apache.beam.sdk.io.TextIOWriteTest.xml [took 4s]

java.nio.file.NoSuchFileException: /tmp/junit8641709517257447332/testwrite7008721698045235481/file.txt-00000-of-00003
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    ...
    at java.nio.file.Files.newBufferedReader(Files.java:2784)
    at org.apache.beam.sdk.io.TextIOWriteTest.readLinesFromFile(TextIOWriteTest.java:512)
    at org.apache.beam.sdk.io.TextIOWriteTest.assertOutputFiles(TextIOWriteTest.java:488)
    at org.apache.beam.sdk.io.TextIOWriteTest.testWriteUnboundedWithCustomBatchParameters(TextIOWriteTest.java:732)

the test is newly added in #31617

Abacn commented 1 week ago

The SamzaRunner seems ignored the sharding. It just outputs a single file /tmp/junit2682757985616873256/testwrite6092321157975729343/file.txt

Amar3tto commented 1 week ago

.take-issue