search

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.89k stars 4.27k forks source link

[collection] environment do not respect flink-conf.yaml. #20709

Open damccorm opened 2 years ago

damccorm commented 2 years ago

When the Flink master is set to -[auto] and- [collection], flink-conf.yaml (from the FLINK_CONF_DIR environment variable, or the --flink-conf-dir option of the job server) is disregarded.

-This can result in some confusing behavior. For example, when [auto] creates a local environment, an empty Configuration() object is used.-

-https://github.com/apache/beam/blob/dd71c9ac472b62ff3598d3db47b0f26985bbc82a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L82-L85-

 

Imported from Jira BEAM-11612. Original Jira may contain additional context. Reported by: ibzib.

Abacn commented 2 years ago

May be fixed by #23635, specifically here: https://github.com/Abacn/beam/blob/2a8077be46af9358990e888283e2a8668002932c/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java#L81 but did not confirm.