apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.88k stars 4.26k forks source link

[Task]: Releases with .dev SdkVersion #27057

Closed shnewto closed 1 year ago

shnewto commented 1 year ago

What needs to happen?

Hello! 👋

I notice that the GO SDK defines its version as .dev of whatever version it's on, SdkVersion = "2.49.0.dev" currently but it also has the .dev in the actual releases, i.e.SdkVersion = "2.48.0.dev" for the 2.48.0 release https://github.com/apache/beam/blob/425186f27dc2eef0e2e45b23cd6eeb7d01ceef20/sdks/go/pkg/beam/core/core.go#L30 (This may be a bug actually, but looking through the file's history it didn't look like un-qualified version numbers were used recently.)

I bring it up because it makes things sort of tricky to use the SDK in base Dataflow Flex templates in GCP. Google says it doesn't support .dev version numbers in Flex templates so when Dataflow jobs are triggered, they fail because Google is looking for 2.48.0.dev (unsupported) instead of 2.48.0. I have a couple workarounds, one is to vendor the source code and change the SdkVersion. The other that seems to work is to pass environment_config='apache/beam_go_sdk:2.48.0 to short circuit whatever's happening in the gcr.io/dataflow-templates-base/go-template-launcher-base image.

I'd be happy to make a PR if there's an appropriate thing to change in docs or some pre-release tests? But it seems like this might just be a process thing for the next 2.49.0 release.

Thanks for all the work that goes into Beam!

*EDIT: I forgot to even mention that the SDK version linked above is being used to define DefaultDockerImage which is actually what's making GCP hiccup I think https://github.com/apache/beam/blob/425186f27dc2eef0e2e45b23cd6eeb7d01ceef20/sdks/go/pkg/beam/core/core.go#L33

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

lostluck commented 1 year ago

It turns out there was a mistake in the release process in the move to using Actions to do parts of it. That was tracked and fixed in https://github.com/apache/beam/issues/27064 as a result there's now a v2.48.2 patch release for the Go SDK, to resolve that issue.

It will still use the 2.48.0 containers by default however, since those were not broken, and were not rebuilt and tagged as a part of this process.

Ironically, due to how Go Modules versions are pushed, there's no test in the world that would have caught this before the tag, since the RC tags were correct. The problem was the RC tag wasn't used in making the Go SDK release tag!

So, @shnewto I'll be closing this issue, since I think the v2.48.2 patch should resolve this for you. But please do reopen if it is causing issues with flex templates.