[Bug]: Portable Spark 3 Streaming runner appears to have GBK bugs, cannot execute Redistribute composite correctly - Githubissues

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

https://beam.apache.org/

Apache License 2.0

7.75k stars 4.21k forks source link

[Bug]: Portable Spark 3 Streaming runner appears to have GBK bugs, cannot execute Redistribute composite correctly #31234

Open kennknowles opened 3 months ago

kennknowles commented 3 months ago

What happened?

See e.g. https://ge.apache.org/s/fqs6k37b63yko/tests/task/:runners:spark:3:job-server:validatesPortableRunnerStreaming/details/org.apache.beam.sdk.transforms.RedistributeTest?top-execution=1

There is no output from the test.

Issue Priority

Priority: 2 (default)

Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[X] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

kennknowles commented 3 months ago

I notice that the portable Spark runner sickbays tests that would probably catch this too:

excludeTestsMatching 'org.apache.beam.sdk.transforms.GroupByKeyTest$WindowTests'

kennknowles commented 3 months ago

Ah and the equivalent ReshuffleTest

Zawa-ll commented 1 month ago

.take-issue