google / fhir-data-pipes

A collection of tools for extracting FHIR resources and analytics services on top of that data.
https://google.github.io/fhir-data-pipes/
Apache License 2.0
142 stars 82 forks source link

Deprecation of `maxWorkers` param #951

Closed chandrashekar-s closed 5 months ago

chandrashekar-s commented 5 months ago

Description of what I changed

Fixes #948

Deprecated the maxWorkers param which was used to set the maxParallelism param in FlinkPipelineOptions. This param only gets used in a Streaming environment, since the pipelines doesn't use any streaming currently this parameter is not used and hence deprecated.

This PR also contains changes which were missed in the PR #935.

E2E test

Relined on e2e. Also, locally verified the Parquet Row Group Sizes for the newly created files.

TESTED:

Please replace this with a description of how you tested your PR beyond the automated e2e/unit tests.

Checklist: I completed these to help reviewers :)

chandrashekar-s commented 5 months ago

@bashir2 This PR contains changes to deprecate the maxWorkers config parameters and also the few changes which were missed for the PR #935.

bashir2 commented 5 months ago

I am going to merge this as there are other dependabot PRs that I want to merge (but I don't want us to wait for another e2e run of this, so I merge this first).

chandrashekar-s commented 5 months ago

I am going to merge this as there are other dependabot PRs that I want to merge (but I don't want us to wait for another e2e run of this, so I merge this first).

Thanks @bashir2