Abacn
commented
1 year ago The performance of WindowInto may worth investigation as I noticed that Python text IO write has worse performance than Java SDK, and the slowest DoFn is WindowInto(GlobalWindows()):
Java metrics: http://104.154.241.245/d/bnlHKP3Wz/java-io-it-tests-dataflow?orgId=1&viewPanel=4 Python metrics: http://104.154.241.245/d/gP7vMPqZz/python-io-it-tests-dataflow?orgId=1&viewPanel=5
Java Read ~20s; Java Write ~30s; Python Read ~100s; Python Write 270s
Two noticable difference from job graph
- Python generate records is much slower than java generatesequence. apache_beam.testing.synthetic_pipeline.SyntheticSource is slow (wall time 14 min 10 sec). Nevertheless this part does not count in write_time metrics.
- apache_beam.transforms.core.WindowIntoFn is slow (wall time 14 min 49 sec).
The Java write pipeline graph looks like this:
The Python write pipeline graph looks like this:
Add microbenchmarks for the windowinto transform:
R: [~tvalentyn]
Imported from Jira BEAM-4855. Original Jira may contain additional context. Reported by: matthiasml6.