[Feature Request]: Add Postcommit Dataflow ARM tests for Java and Python - Githubissues

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

https://beam.apache.org/

Apache License 2.0

7.81k stars 4.23k forks source link

[Feature Request]: Add Postcommit Dataflow ARM tests for Java and Python #28105

Open celeste-zeng opened 1 year ago

celeste-zeng commented 1 year ago

What would you like to happen?

Currently only Go has a PostCommit test suite on Dataflow ARM: https://github.com/apache/beam/blob/master/.github/workflows/beam_PostCommit_Go_Dataflow_ARM.yml.

We want to have PostCommit ARM test suites for Python and Java that run not only wordcount but also other tests on Dataflow runner.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

[X] Component: Python SDK
[ ] Component: Java SDK
[X] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

celeste-zeng commented 1 year ago

The Java11 PostCommit Dataflow on x86 counterpart, such as job_PostCommit_Java_Dataflow_Examples_Java11.groovy, also only runs https://github.com/apache/beam/blob/01cd45390babcc5d0570affa73d604f5077a6d15/runners/google-cloud-dataflow-java/examples/build.gradle#L69C8-L70C46 wordcount tests, not sure if we should add more tests to it.

celeste-zeng commented 1 year ago

Tests to be added for Java PostCommit: https://github.com/apache/beam/blob/7a421c4356a4326461f78231630575666598a8ff/runners/google-cloud-dataflow-java/build.gradle#L606 (remember not to exclude the wordcount).