We are happy to present the new 2.53.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.
The Python SDK now type checks collections.abc.Collections types properly. Some type hints that were erroneously allowed by the SDK may now fail. (#29272)
Running multi-language pipelines locally no longer requires Docker.
Instead, the same (generally auto-started) subprocess used to perform the
expansion can also be used as the cross-language worker.
Framework for adding Error Handlers to composite transforms added in Java (#29164).
Python 3.11 images now include google-cloud-profiler (#29561).
Adding support for LowCardinality (Java) (#29533).
New Features / Improvements
The Python SDK now type checks collections.abc.Collections types properly. Some type hints that were erroneously allowed by the SDK may now fail. (#29272)
Running multi-language pipelines locally no longer requires Docker.
Instead, the same (generally auto-started) subprocess used to perform the
expansion can also be used as the cross-language worker.
Framework for adding Error Handlers to composite transforms added in Java (#29164).
Python 3.11 images now include google-cloud-profiler (#29561).
Euphoria DSL is deprecated and will be removed in a future release (not before 2.56.0) (#29451)
Bugfixes
(Python) Fixed sporadic crashes in streaming pipelines that affected some users of 2.47.0 and newer SDKs (#27330).
(Python) Fixed a bug that caused MLTransform to drop identical elements in the output PCollection (#29600).
[2.52.0] - 2023-11-17
Highlights
Previously deprecated Avro-dependent code (Beam Release 2.46.0) has been finally removed from Java SDK "core" package.
Please, use beam-sdks-java-extensions-avro instead. This will allow to easily update Avro version in user code without
potential breaking changes in Beam "core" since the Beam Avro extension already supports the latest Avro versions and
should handle this. (#25252).
Publishing Java 21 SDK container images now supported as part of Apache Beam release process. (#28120)
Direct Runner and Dataflow Runner support running pipelines on Java21 (experimental until tests fully setup). For other runners (Flink, Spark, Samza, etc) support status depend on runner projects.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps org.apache.beam:beam-sdks-java-google-cloud-platform-bom from 2.51.0 to 2.53.0.
Release notes
Sourced from org.apache.beam:beam-sdks-java-google-cloud-platform-bom's releases.
... (truncated)
Changelog
Sourced from org.apache.beam:beam-sdks-java-google-cloud-platform-bom's changelog.
... (truncated)
Commits
260b554
Set version for 2.53.0 RC21f6d7af
Update google-api-services-storage to match libraries-bom (#29865)5b9bbaf
Remove failing multimap key iteration tests. (#29837)210d0e4
Revert to non-schema-based Sql transform. (#29831)292606b
Fix nil pointer access in logRuntimeDependencies (#29804) (#29806)ea477dd
getProcessingTimesByStep returns a modifiable map (#29786) (#29799)667027d
Fixes breakages of the upgrade feature (#29731) (#29766)03e9036
test revert _beam_schema_proto (#29754) (#29759)f773123
#29760 Only respond to sampling request while data sampling is enabled (#29...7e6fa65
Set Dataflow container to release version.Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase
.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show