apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.89k stars 4.27k forks source link

[Bug]: Cross-language pipeline options are not picked up in Java DoFns #33074

Open ahmedabu98 opened 1 week ago

ahmedabu98 commented 1 week ago

What happened?

I noticed this when testing with Managed BigQueryIO in Python SDK:

options = PipelineOptions([
  "--project=google.com:clouddfe",
  "--region=us-central1",
  "--temp_location=gs://ahmedabualsaud-wordcount/tmp",
])

with beam.Pipeline(options=options) as p:
  (p
   | beam.Create([beam.Row(int=999)])
   | beam.managed.Write("bigquery",
                        config={"table": "google.com:clouddfe.ahmedabualsaud.test_table"},
                        expansion_service="localhost:12345"
                        ))

I would expect these pipeline options to be picked up on Java's side. With some logging, I see the Java SDK recognizing the options at expand time in PTransforms (i.e. expand()), but not at runtime in DoFns (i.e. @ProcessElement).

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components