Update to Protobuf 4.x

[neeme-praks-sympower]

Edit by @ejona86: Protobuf has resolved the incompatibility with protobuf-java 4.x. Use protobuf-java 4.27.4 or later. This is being left open a bit longer for gRPC itself to upgrade, but you don't need to wait for gRPC.

---

Is your feature request related to a problem?

I would like to use the latest released version of Protobuf (4.26.0) but I cannot as gRPC is still using the old version.

Describe the solution you'd like

Update Protobuf dependency and make it work with the new version.

Describe alternatives you've considered

I hoped that the update would be easy (just a version bump) so I tried doing it myself but it resulted in a compilation failure so I'll leave it to you to figure it out.

The failure:

> Task :grpc-protobuf:compileJava FAILED
grpc-java/protobuf/src/main/java/io/grpc/protobuf/StatusProto.java:38: error: cannot access GeneratedMessageV3
          ProtoLiteUtils.metadataMarshaller(com.google.rpc.Status.getDefaultInstance()));
                                                                 ^
  class file for com.google.protobuf.GeneratedMessageV3 not found
grpc-java/protobuf/src/main/java/io/grpc/protobuf/StatusProto.java:171: error: cannot access Builder
        .setCode(status.getCode().value());
        ^
  class file for com.google.protobuf.GeneratedMessageV3$Builder not found
2 errors

[ganadist]

https://protobuf.dev/news/2023-12-05

Also, I'm wondering how this protobuf changes will affect

• gRPC public API changes
• Compatibility with artifacts which were generated by old protobuf/gRPC compilers.

[ejona86]

The protobuf breakage is pretty serious and goes against wisdom like https://jlbp.dev/JLBP-7 . I don't see how the community can absorb it. googleapis libraries are hit even harder than we are, so I think we'd wait until they have an idea how to deal with this (otherwise they couldn't upgrade to newer grpc versions). I don't expect that to be too big of a problem for the community, as I think most protobuf-using projects will be in shock.

I'm in discussions with others, but I'd expect the process to figure this out to take months.

[augi]

After updating to Protobuf 4.26.0, we observe this issue:

java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
    at io.grpc.health.v1.HealthGrpc.getCheckMethod(HealthGrpc.java:38)
    at io.grpc.health.v1.HealthGrpc.getServiceDescriptor(HealthGrpc.java:413)
    at io.grpc.health.v1.HealthGrpc.bindService(HealthGrpc.java:350)
    at io.grpc.health.v1.HealthGrpc$HealthImplBase.bindService(HealthGrpc.java:169)

The announcement of the Protobuf release contains this: [Java] The base class for generated messages will be GeneratedMessage, not GeneratedMessageV3.

So I guess this has to be adjusted.

[suztomo]

@neeme-praks-sympower Does any library or dependency require you to upgrade Protobuf runtime version to 4.x?

[be-hase]

It seems that proto-google-common-protos, which grpc-protobuf depends on, is not yet ver4 compliant. https://github.com/googleapis/sdk-platform-java/tree/main/java-common-protos

[krickert]

Why couldn't grpc just bring back generated message v3? The classes appear almost identical, I suppose it can just extend it?

I tried that myself - it somewhat worked. I got past the generated message v3 error, but I was getting serializaiton errors everywhere. I think they may have changed the serializaiton spec too?

Anyway, I'm looking forward to this fix. I might try this again today.

[neeme-praks-sympower]

@neeme-praks-sympower Does any library or dependency require you to upgrade Protobuf runtime version to 4.x?

No. We just try to keep our dependencies up-to-date.

[suztomo]

@neeme-praks-sympower Thank you.

[ummels]

I've opened an issue at Protobuf for restoring GeneratedMessageV3 as an alias for GeneatedMessage.

[krickert]

I've opened an issue at Protobuf for restoring GeneratedMessageV3 as an alias for GeneatedMessage.

I didn't think you're going to win that though

It was a major upgrade with breaking changes. They deprecated V3 awhile ago, better to fix instead of bringing it back. It'll take some time, but GRPC isn't always caught up to the latest protocol buffers.

[snazy]

The protobuf v4 release effectively breaks a ton of projects. Sure, it's a breaking change - but it is a breaking change without a migration path. The protobuf code generators generated GeneratedMessageV3 and that is now gone - so the tons of existing and working projects that use the protobuf code generators will stop working with protobuf v4.

IMHO, a breaking change must first provide a properly communicated migration path and then give all users and projects enough time to adopt.

Bringing back GeneratedMessageV3 as a migration path SGTM - just not sure how good that'll work.

[ummels]

I'm in for a fight. :joy:

Anyway, if they deprecated V3 a while ago, how come code generated with protoc 3.25.3 still relies on GeneratedMessageV3? Doesn't seem like a smooth transition to me. :thinking:

[ejona86]

Quick update: protobuf understands the problem and are considering solutions, and they've said as much publicly. I expect a few more weeks to settle on the approach and then some time to make sure the details are right.

For those watching at home and want to be prepared, use protobuf-java 3.25 and protoc 25. Also, if you have any non-generated code using GeneratedMessageV3, replace those usages with Message. Even if GeneratedMessageV3 is added back to protobuf 4.x the newer generated code won't implement it. It was never intended to be referenced outside of generated code. I'm not aware of any reason to reference GeneratedMessageV3 in hand-written code, so it is hopefully easy to avoid.

(I'm assuming there's no references to GeneratedMessage and protoc 2 since 3.0 came out in 2016. Any expected solution is expected to drop support for the 2.x generated code.)

[ejona86]

See https://github.com/protocolbuffers/protobuf/issues/16452#issuecomment-2076360676 for an update from protobuf. There's going to be a new 3.x release that will be compatible with 4.x. The ecosystem would upgrade to that new 3.x release before upgrading to 4.x. Once that release exists, gRPC will try to quickly pick it up, and might even backport the version bump (not something we typically do, but this is serious enough to probably be worth the risk).

[ejona86]

Protobuf 27.2 (protobuf-java 4.27.2) has restored compatibility with (at least some later versions of) the 3.x generated code. https://github.com/protocolbuffers/protobuf/releases/tag/v27.2

grpc-java should now be compatible with protobuf-java 4.x. The Protobuf change was tested against 3.25.0, which means at least grpc-java 1.61+ should be compatible.

[krickert]

Protobuf 27.2 (protobuf-java 4.27.2) has restored compatibility with (at least some later versions of) the 3.x generated code. https://github.com/protocolbuffers/protobuf/releases/tag/v27.2

grpc-java should now be compatible with protobuf-java 4.x. The Protobuf change was tested against 3.25.0, which means at least grpc-java 1.61+ should be compatible.

That's great news. I blame dependabot for making this bother me, but it wasn't that long of a wait at all.

[be-hase]

FYI: It seems that there are still some issues remaining with protobuf-java.

https://github.com/protocolbuffers/protobuf/issues/17247

[ejona86]

Appears the further issues may be resolved soon. Testing with the 4.28.0-RC3 looked good to me. https://github.com/protocolbuffers/protobuf/issues/17247#issuecomment-2305918282

[ejona86]

Protobuf 27.4 was released yesterday. We're targeting upgrading in 1.67.

[ascopes]

do we have a timescale for this, out of curiosity?

[ejona86]

@ascopes, "We're targeting upgrading in 1.67". The schedule for 1.67 is in the milestone (Sept 17th). But you don't have to wait on us. You can upgrade protobuf today in your own app.

[ascopes]

@ejona86 thanks, was asking as on the most recent version of protobuf and gRPC, the introspection service for gRPC-java was still failing due to the use of v3 APIs within protobuf.

[ejona86]

@ascopes, use protobuf-java 4.27.4 or later.

[ascopes]

thanks, shall give it a try.