We are happy to present the new 2.58.1 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.
New Features / Improvements
Fixed issue where KafkaIO Records read with ReadFromKafkaViaSDF are redistributed and may contain duplicates regardless of the configuration. This affects Java pipelines with Dataflow v2 runner and xlang pipelines reading from Kafka, (#32196)
Known Issues
Large Dataflow graphs using runner v2, or pipelines explicitly enabling the upload_graph experiment, will fail at construction time (#32159).
Python pipelines that run with 2.53.0-2.58.0 SDKs and read data from GCS might be affected by a data corruption issue (#32169). The issue will be fixed in 2.59.0 (#32135). To work around this, update the google-cloud-storage package to version 2.18.2 or newer.
According to git shortlog, the following people contributed to the 2.58.1 release. Thank you to all contributors!
Danny McCormick
Sam Whittle
Beam 2.58.0 release
We are happy to present the new 2.58.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.
Fixed issue where KafkaIO Records read with ReadFromKafkaViaSDF are redistributed and may contain duplicates regardless of the configuration. This affects Java pipelines with Dataflow v2 runner and xlang pipelines reading from Kafka, (#32196)
Known Issues
Large Dataflow graphs using runner v2, or pipelines explicitly enabling the upload_graph experiment, will fail at construction time (#32159).
Python pipelines that run with 2.53.0-2.58.0 SDKs and read data from GCS might be affected by a data corruption issue (#32169). The issue will be fixed in 2.59.0 (#32135). To work around this, update the google-cloud-storage package to version 2.18.2 or newer.
[2.58.0] - 2024-08-06
Highlights
Support for Solace source (SolaceIO.Read) added (Java) (#31440).
New Features / Improvements
Multiple RunInference instances can now share the same model instance by setting the model_identifier parameter (Python) (#31665).
Added options to control the number of Storage API multiplexing connections (#31721)
[BigQueryIO] Better handling for batch Storage Write API when it hits AppendRows throughput quota (#31837)
[IcebergIO] All specified catalog properties are passed through to the connector (#31726)
Removed a 3rd party LGPL dependency from the Go SDK (#31765).
Support for MapState and SetState when using Dataflow Runner v1 with Streaming Engine (Java) ([#18200])
Breaking Changes
[IcebergIO] IcebergCatalogConfig was changed to support specifying catalog properties in a key-store fashion (#31726)
[SpannerIO] Added validation that query and table cannot be specified at the same time for SpannerIO.read(). Previously withQuery overrides withTable, if set (#24956).
Bugfixes
[BigQueryIO] Fixed a bug in batch Storage Write API that frequently exhausted concurrent connections quota (#31710)
Fixed a logging issue where Python worker dependency installation logs sometimes were not emitted in a timely manner (#31977)
Known Issues
Large Dataflow graphs using runner v2, or pipelines explicitly enabling the upload_graph experiment, will fail at construction time (#32159).
Python pipelines that run with 2.53.0-2.58.0 SDKs and read data from GCS might be affected by a data corruption issue (#32169). The issue will be fixed in 2.59.0 (#32135). To work around this, update the google-cloud-storage package to version 2.18.2 or newer.
[KafkaIO] Records read with ReadFromKafkaViaSDF are redistributed and may contain duplicates regardless of the configuration. This affects Java pipelines with Dataflow v2 runner and xlang pipelines reading from Kafka, (#32196)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
dependabot[bot]
commented
2 days ago
Bumps org.apache.beam:beam-sdks-java-google-cloud-platform-bom from 2.51.0 to 2.58.1.
Release notes
Sourced from org.apache.beam:beam-sdks-java-google-cloud-platform-bom's releases.
... (truncated)
Changelog
Sourced from org.apache.beam:beam-sdks-java-google-cloud-platform-bom's changelog.
... (truncated)
Commits
414bc20Set version for 2.58.1 RC157e5a46Cherry pick (#32202)5e69723Update dataflow container versions (#32203)7fbbe0bAdds ordered list user state support to fnapi accessor cache. (#31923) (#32013)bb828dcRevert "[Cherrypick] Update Build Wheels to only build once on RCs (#32009)" ...0a4bdc3Update Build Wheels to only build once on RCs (#32009) (#32010)cdc19d0[yaml] Fix yaml provider schema validation and merging (#31974) (#31984)3ec5643Dedup Lineage and getTableToExtract call in BigQuerySourceBase (#31960) (#31972)6e19e38Use buffered loggers that periodically flush. (#31978)eca36f6Switch to use self-hosted runner for build_wheels action (#31866) (#31916)Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show