search

googleapis / go-genproto

Generated code for Google Cloud client libraries.
https://pkg.go.dev/google.golang.org/genproto
Apache License 2.0
591 stars 188 forks source link

New submodules: upgrade before 2024-03-01 to avoid ambiguous import errors with google.golang.org/genproto #1015

Open quartzmo opened 1 year ago

quartzmo commented 1 year ago

TL;DR

Upgrade to a version of google.golang.org/genproto at or above version v0.0.0-20230526161137-0005af68ea54 with the following command:

go get google.golang.org/genproto@latest
go mod tidy

Doing this before the 9-month deadline of 2024-03-01 will avoid "ambiguous import" errors.

Background

Many of the packages in the google.golang.org/genproto module represent API surfaces belonging to Google Cloud. Since approximately the v0.0.0-20220921223823-23cae91e6737 release of google.golang.org/genproto, a change in #901 introduced dependencies on virtually every submodule in cloud.google.com/go. This change moved type definitions for Google Cloud client libraries to cloud.google.com/go, greatly improving usability of the latter. To maintain compatibility with existing code, aliases were introduced into google.golang.org/genproto that pointed to the new sources of truth in cloud.google.com/go. This added the dependencies on cloud modules.

Even though most packages in google.golang.org/genproto are Google Cloud, a few packages provide simple “common” types used in projects unrelated to Google Cloud. For users of the simpler packages, this commingled existence resulted in a large number of extra cloud modules appearing in the dependency graph. This can trigger alarms in automatic dependency checks, false positives in vulnerability checks, and adds constraints to minimum version selection when resolving acceptable versions of other dependencies.

New submodules in this repo

To improve usability for consumers that do not depend on Google Cloud modules, packages providing widely used "common" types will be published as submodules. The following packages do not depend on cloud types and are now isolated in new non-cloud submodules:

The following subpackages of google.golang.org/genproto/googleapis/api reference cloud types and are now isolated from the api package as new cloud submodules:

To ease migration, these submodules initially have a “back-ref” dependency on the google.golang.org/genproto parent. After a 9-month period, this "back-ref" dependency will be removed, making it possible to once again depend on the simpler packages of this project without adding cloud modules to the dependency graph.

Avoiding ambiguous imports for submodules

The root package docs for google.golang.org/genproto state:

IMPORTANT This repository is currently experimental. The structure of the contained packages is subject to change. Please see the original source repositories (listed below) to find out the status of each protocol buffer's associated service.

In addition to this warning, the pseudo version numbering of the module:

Signals that the module is still in development and unstable. This release carries no backward compatibility or stability guarantees.

However, despite these warnings, we know that critical infrastructure depends on many of the packages in google.golang.org/genproto. To avoid "ambiguous import" errors for consumers, we are adding a “back-ref” dependency on the google.golang.org/genproto parent to each new submodule for 9 months to allow a seamless transition.

After the 9-month deadline of 2024-03-01 has passed, the “back-ref” dependency on the google.golang.org/genproto parent will be removed from each new submodule. Client code still using a version of the google.golang.org/genproto parent earlier than v0.0.0-20230525234044-86246a831fb4 may then encounter "ambiguous import" errors when importing packages that have been moved to submodules. Those errors can be resolved by updating to a version at or above version v0.0.0-20230526161137-0005af68ea54 with the following command:

go get google.golang.org/genproto@latest
go mod tidy

References

smocherla-brex commented 9 months ago

We're going to migrate but curious regarding the behavior after 2024-03-01

Client code still using a version of the google.golang.org/genproto parent earlier than v0.0.0-20230525234044-86246a831fb4 may then encounter "ambiguous import" errors when importing packages that have been moved to submodules

If we were using offical Go modules proxy which caches the modules indefinitely (from my understanding), will this still error out (or will it be removed from the cache as well)?

liggitt commented 9 months ago

It isn't a matter of the old module versions being removed or becoming unavailable, they would still be available via module proxy or git sha.

The "ambiguous import" issue happens when some dependencies in your dependency graph refer to a version of genproto prior to the carveout, some things refer to a version after the carveout along with some of the submodules, and dependency resolution can't figure out which module is supposed to be providing the package.

Making sure you require a version of genproto after the carveout makes it unambiguous which module provides that package.

quartzmo commented 6 months ago

Closing. as the major projects targeted by this effort have been upgraded to the new submodules.

kb-sp commented 5 months ago

If you're here because your builds are now breaking within your Google framework dependencies, this is a good example of what happens when a Go project doesn't practice standard Go versioning.

EDIT: In the end, my issue was a dependency on the old api/idtoken that, in some circumstances, created dependency resolution ambiguity. Moving to a combination of cloud.google.com/go/auth/credentials/idtoken and cloud.google.com/go/auth.

While this repository is marked as experimental (Alpha? Beta?), there are versioned, production Google modules that reference this repository via un-versioned tags. IMHO, there's cognitive dissonance here, where either the other modules are actually tainted to Alpha, or this repository really isn't.

ldemailly commented 1 month ago

Just to point out in case it helps someone, that I had a build suddenly break out of the blue... (way past the deadline) and the fix was https://github.com/fortio/fortiotel/pull/254/commits/e5572ad176041f84c638d4a9a8a40b7011ca58b2

ie

go get google.golang.org/genproto@latest

but not do go mod tidy which removes somehow the 

    google.golang.org/genproto v0.0.0-20240924160255-9d4c2d233b61 // indirect

entry

ldemailly commented 1 month ago

the above (go mod tidy removing the entry) is a problem because it means every dependabot PR fails... any idea what the correct fix is?

codyoss commented 1 month ago

@ldemailly can you give a small reproducer of this problem with a go.mod. Likely the solution is to bump some other out of date dependency.

ldemailly commented 1 month ago

@ldemailly can you give a small reproducer of this problem with a go.mod. Likely the solution is to bump some other out of date dependency.

Thanks! yes:

git clone https://github.com/fortio/fortiotel
cd fortiotel
go build . # all good/fine
go mod tidy
git diff
go build .  # error as below and this original ticket

diff is 

13:57:13 main fortiotel$ go mod tidy
13:57:18 main fortiotel$ git diff
diff --git a/go.mod b/go.mod
index d54dab6..3be40ca 100644
--- a/go.mod
+++ b/go.mod
@@ -46,7 +46,6 @@ require (
        golang.org/x/net v0.29.0 // indirect
        golang.org/x/sys v0.25.0 // indirect
        golang.org/x/text v0.18.0 // indirect
-       google.golang.org/genproto v0.0.0-20240924160255-9d4c2d233b61 // indirect
        google.golang.org/genproto/googleapis/api v0.0.0-20240903143218-8af14fe29dc1 // indirect
        google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
        google.golang.org/grpc v1.67.1 // indirect

error after tidy is

$ go build .
../../../go/pkg/mod/google.golang.org/grpc@v1.67.1/status/status.go:35:2: ambiguous import: found package google.golang.org/genproto/googleapis/rpc/status in multiple modules:
    google.golang.org/genproto v0.0.0-20230216225411-c8e22ba71e44 (/Users/dl/go/pkg/mod/google.golang.org/genproto@v0.0.0-20230216225411-c8e22ba71e44/googleapis/rpc/status)
    google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 (/Users/dl/go/pkg/mod/google.golang.org/genproto/googleapis/rpc@v0.0.0-20240903143218-8af14fe29dc1/status)
../../../go/pkg/mod/github.com/grpc-ecosystem/grpc-gateway/v2@v2.22.0/runtime/handler.go:13:2: ambiguous import: found package google.golang.org/genproto/googleapis/api/httpbody in multiple modules:
    google.golang.org/genproto v0.0.0-20230216225411-c8e22ba71e44 (/Users/dl/go/pkg/mod/google.golang.org/genproto@v0.0.0-20230216225411-c8e22ba71e44/googleapis/api/httpbody)
    google.golang.org/genproto/googleapis/api v0.0.0-20240903143218-8af14fe29dc1 (/Users/dl/go/pkg/mod/google.golang.org/genproto/googleapis/api@v0.0.0-20240903143218-8af14fe29dc1/httpbody)
../../../go/pkg/mod/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc@v1.30.0/client.go:12:2: ambiguous import: found package google.golang.org/genproto/googleapis/rpc/errdetails in multiple modules:
    google.golang.org/genproto v0.0.0-20230216225411-c8e22ba71e44 (/Users/dl/go/pkg/mod/google.golang.org/genproto@v0.0.0-20230216225411-c8e22ba71e44/googleapis/rpc/errdetails)
    google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 (/Users/dl/go/pkg/mod/google.golang.org/genproto/googleapis/rpc@v0.0.0-20240903143218-8af14fe29dc1/errdetails)
codyoss commented 1 month ago

I have filed https://github.com/envoyproxy/go-control-plane/issues/1014. In the meantime another workaround would be to underscore import genproto in your project so that you can control the version of the dependency and there is no ambiguity.

ldemailly commented 1 month ago

turns out... it's the go.work or something related to the simple/ sub directory (btw trying to _ import didn't work because there is nothing at the top and even trying to read say 

    protobuf "google.golang.org/genproto/protobuf/api"

 protobuf.File_google_protobuf_api_proto

also somehow left it in indirect

ldemailly commented 1 month ago

So in short, sorry for the noise yet maybe that oddity that a sub project go.mod can cause that issue might help someone else

ldemailly commented 1 month ago

well it's actually not fixed... go mod tidy in simple if they all use the same toolchain does remove it... thankfully that one isn't done by dependabot and rots but... something fishy is going on

bmulder-innoseis commented 2 weeks ago

well it's actually not fixed... go mod tidy in simple if they all use the same toolchain does remove it... thankfully that one isn't done by dependabot and rots but... something fishy is going on

@ldemailly I don't understand what you mean and I have the same issue. What did you do to work around it?

ldemailly commented 2 weeks ago

@ldemailly I don't understand what you mean and I have the same issue. What did you do to work around it?

the workaround I have is to have a subdirectory with its own go.mod with the added indirect require - having that allows the real go.mod to work and survive go mod tidy. it's a horrible workaround and I don't understand why/how it works (see fortio/fortiotel repo)

bmulder-innoseis commented 2 weeks ago

the workaround I have is to have a subdirectory with its own go.mod with the added indirect require

Oh dear... Anyway, thanks I guess

Maybe someone can reopen this in the meantime? Or is there another ticket for this?

quartzmo commented 1 week ago

See https://github.com/envoyproxy/go-control-plane/issues/1014.