[Failing Test]: Many IO_Direct Precommits flaky due to Jacoco abort "No space left on the disk"

Abacn commented 1 year ago

What happened?

Many IO_Direct Precommit test becomes very flaky since Friday, Apr 7 2023. Looking at the logs the tests actually succeeded, but failing to report the code coverage:

05:12:59 [JaCoCo plugin] Saving matched class directories for class-pattern: **/classes: ERROR: Step ‘Record JaCoCo coverage report’ aborted due to exception: 
05:12:59 java.nio.file.FileSystemException: /home/jenkins/jenkins-home/jobs/beam_PreCommit_Java_Tika_IO_Direct_Cron/builds/489/jacoco/classes/groovy/main/org/apache/beam/gradle/BeamModulePlugin$_apply_closure10$_closure76.class: No space left on device
05:12:59    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
...
05:12:59 Caused: java.io.IOException: Failed to extract /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Tika_IO_Direct_Cron/src/buildSrc/build/classes/**/*.class
05:12:59    at hudson.FilePath.readFromTar(FilePath.java:2992)

This is related to the recent change that re-enabling code coverage for IOs: #25956

I'm going to disable the code coverage again and leave this task open to investigate what is exactly the root cause. To mitigate the pressure on CI, probably we need an independent code coverage PostCommit instead of many precommits.

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

[ ] Component: Python SDK
[X] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[X] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

mosche commented 1 year ago

This isn't limited to IOs, "ordinary" Java precommit runs fail for the same reason, e.g. https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/6013/console

Abacn commented 1 year ago

This has been mitigated by multiple efforts: #26199 #26224

apache / beam