We are happy to present the new 2.35.0 release of Apache Beam.
This release includes both improvements and new functionality.
See the download page for this release.
MultiMap side inputs are now supported by the Go SDK (BEAM-3293).
Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (BEAM-12522).
Upgrades Log4j version used in test suites (Apache Beam testing environment only, not for end user consumption) to 2.17.0(BEAM-13434).
Note that Apache Beam versions do not depend on the Log4j 2 dependency (log4j-core) impacted by CVE-2021-44228.
However we urge users to update direct and indirect dependencies (if any) on Log4j 2 to the latest version by updating their build configuration and redeploying impacted pipelines.
I/Os
We changed the data type for ranges in JdbcIO.readWithPartitions from int to long (BEAM-13149).
This is a relatively minor breaking change, which we're implementing to improve the usability of the transform without increasing cruft.
This transform is relatively new, so we may implement other breaking changes in the future to improve its usability.
Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (BEAM-12522).
New Features / Improvements
Added custom delimiters to Python TextIO reads (BEAM-12730).
Added escapechar parameter to Python TextIO reads (BEAM-13189).
Splittable reading is enabled by default while reading data with ParquetIO (BEAM-12070).
DoFn Execution Time metrics added to Go (BEAM-13001).
Cross-bundle side input caching is now available in the Go SDK for runners that support the feature by setting the EnableSideInputCache hook (BEAM-11097).
Upgraded the GCP Libraries BOM version to 24.0.0 and associated dependencies (BEAM-11205). For Google Cloud client library versions set by this BOM,
see this table.
Removed avro-python3 dependency in AvroIO. Fastavro has already been our Avro library of choice on Python 3. Boolean use_fastavro is left for api compatibility, but will have no effect.(BEAM-13016).
MultiMap side inputs are now supported by the Go SDK (BEAM-3293).
Remote packages can now be downloaded from locations supported by apache_beam.io.filesystems. The files will be downloaded on Stager and uploaded to staging location. For more information, see BEAM-11275
Breaking Changes
A new URN convention was adopted for cross-language transforms and existing URNs were updated. This may break advanced use-cases, for example, if a custom expansion service is used to connect diffrent Beam Java and Python versions. (BEAM-12047).
The upgrade to Calcite 1.28.0 introduces a breaking change in the SUBSTRING function in SqlTransform, when used with the Calcite dialect (BEAM-13099, CALCITE-4427).
ListShards (with DescribeStreamSummary) is used instead of DescribeStream to list shards in Kinesis streams (AWS SDK v2). Due to this change, as mentioned in AWS documentation, for fine-grained IAM policies it is required to update them to allow calls to ListShards and DescribeStreamSummary APIs. For more information, see Controlling Access to Amazon Kinesis Data Streams (BEAM-13233).
Deprecations
Non-splittable reading is deprecated while reading data with ParquetIO (BEAM-12070).
MultiMap side inputs are now supported by the Go SDK (BEAM-3293).
Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (BEAM-12522).
Upgrades Log4j version used in test suites (Apache Beam testing environment only, not for end user consumption) to 2.17.0(BEAM-13434).
Note that Apache Beam versions do not depend on the Log4j 2 dependency (log4j-core) impacted by CVE-2021-44228.
However we urge users to update direct and indirect dependencies (if any) on Log4j 2 to the latest version by updating their build configuration and redeploying impacted pipelines.
I/Os
We changed the data type for ranges in JdbcIO.readWithPartitions from int to long (BEAM-13149).
This is a relatively minor breaking change, which we're implementing to improve the usability of the transform without increasing cruft.
This transform is relatively new, so we may implement other breaking changes in the future to improve its usability.
Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (BEAM-12522).
New Features / Improvements
Added custom delimiters to Python TextIO reads (BEAM-12730).
Added escapechar parameter to Python TextIO reads (BEAM-13189).
Splittable reading is enabled by default while reading data with ParquetIO (BEAM-12070).
DoFn Execution Time metrics added to Go (BEAM-13001).
Cross-bundle side input caching is now available in the Go SDK for runners that support the feature by setting the EnableSideInputCache hook (BEAM-11097).
Upgraded the GCP Libraries BOM version to 24.0.0 and associated dependencies (BEAM-11205). For Google Cloud client library versions set by this BOM,
see this table.
Removed avro-python3 dependency in AvroIO. Fastavro has already been our Avro library of choice on Python 3. Boolean use_fastavro is left for api compatibility, but will have no effect.(BEAM-13016).
MultiMap side inputs are now supported by the Go SDK (BEAM-3293).
Remote packages can now be downloaded from locations supported by apache_beam.io.filesystems. The files will be downloaded on Stager and uploaded to staging location. For more information, see BEAM-11275
Breaking Changes
A new URN convention was adopted for cross-language transforms and existing URNs were updated. This may break advanced use-cases, for example, if a custom expansion service is used to connect diffrent Beam Java and Python versions. (BEAM-12047).
The upgrade to Calcite 1.28.0 introduces a breaking change in the SUBSTRING function in SqlTransform, when used with the Calcite dialect (BEAM-13099, CALCITE-4427).
ListShards (with DescribeStreamSummary) is used instead of DescribeStream to list shards in Kinesis streams (AWS SDK v2). Due to this change, as mentioned in AWS documentation, for fine-grained IAM policies it is required to update them to allow calls to ListShards and DescribeStreamSummary APIs. For more information, see Controlling Access to Amazon Kinesis Data Streams (BEAM-13233).
Deprecations
Non-splittable reading is deprecated while reading data with ParquetIO (BEAM-12070).
Bugfixes
Properly map main input windows to side input windows by default (Go)
(BEAM-11087).
Fixed data loss when writing to DynamoDB without setting deduplication key names (Java)
(BEAM-13009).
Go SDK Examples now have types and functions registered. (Go) (BEAM-5378)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
dependabot[bot]
commented
2 years ago
Bumps apache-beam[gcp] from 2.33.0 to 2.35.0.
Release notes
Sourced from apache-beam[gcp]'s releases.
... (truncated)
Changelog
Sourced from apache-beam[gcp]'s changelog.
... (truncated)
Commits
a3400e3Set version for 2.35.0 RC87eb7fd0Cherrypick a few more Log4j fixes. #163529b66a80[BEAM-13490] Bump log4j version to 2.17.0 (#16276)d7fa66dUse a patched shadow 6.1.0 plugin using Log4j 2.16.0 (#16269)1ff38aa[release-2.35.0][BEAM-13499] Add warning about log4j to hcatalog javadoc #16311f615f28Add warning to hcatalog javadoc6106fbc[BEAM-13434] Pin transitive log4j dependencies to 2.17.0 in ...:hcatalog modu...af05662[BEAM-13388] [BEAM-13052] Cherry-pick ProtoCoder implementation onto 2.35.0 r...a45d570[BEAM-13496] [cp] Upgrade Flink runner to include log4j patches. #163007e25626[release-2.35.0][BEAM-13434] Bump solr to 8.11.1 #16299Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)