Unable to use lite version - multiple versions of class

[ilker-aktuna]

I am trying to upgrade my Android app so that it targets a newer Android SDK (33 or 34)

I realized that if I continue using protobuf 3.6.1 , it does not compile with errors such as: "memoizedserializedsize has private access in generatedmessagelite int size = memoizedserializedsize"

So I decided to update used protobuf libraries. Ended up with protobuf 3.22.3 This one compiles well and the app runs fine. But now if I attempt to build an apk or app bundle with Android Studio, it does not build because of multiple verisons of a class.

The full error message is:

Type com.google.protobuf.Any$1 is defined multiple times: C:\Users\ilker.gradle\caches\transforms-3\f587bdbbf1ee885f5ee14d4cd9b8bf8b\transformed\jetified-protobuf-javalite-3.22.3.jar:com/google/protobuf/Any$1.class, C:\Development\workspace\android_studio\VectorCTRLPRO\app\build\intermediates\javac\release\classes\com\google\protobuf\Any$1.class

If I build a "debug" package it compiles well, but a "release" package build throws the error above.

My build.gradle related to protobuf .is like this:

    implementation ('io.grpc:grpc-protobuf-lite:1.56.1') {
        exclude group: 'com.google.protobuf', module: 'protobuf-java'
    }

    protobuf ('com.google.protobuf:protobuf-java:3.22.3') {
        exclude group: 'com.google.protobuf', module: 'protobuf-javalite'
        exclude group: 'com.google.protobuf', module: 'protobuf-java'
    }
    protobuf 'com.google.api.grpc:googleapis-common-protos:0.0.3'

protobuf {
    protoc { artifact = 'com.google.protobuf:protoc:3.22.3' }
    plugins {
        grpc { artifact = 'io.grpc:protoc-gen-grpc-java:1.56.1' }
    }

    generateProtoTasks {
        all().each { task ->
            task.builtins{
                java { option 'lite' }
            }
            task.plugins {
                grpc { option 'lite' }
            }
        }
    }

}

My old build.gradle was different because of the old version of protobuf:

implementation 'io.grpc:grpc-protobuf-lite:1.21.0'
    protobuf 'com.google.protobuf:protobuf-java:3.6.1'
    protobuf 'com.google.api.grpc:googleapis-common-protos:0.0.3'

protobuf {
    protoc { artifact = 'com.google.protobuf:protoc:3.6.1'
        //artifact = 'com.google.protobuf:protoc:3.5.1-1'
    }
    plugins {
        javalite { artifact = "com.google.protobuf:protoc-gen-javalite:3.0.0" }
        grpc {
            artifact = 'io.grpc:protoc-gen-grpc-java:1.21.0' // CURRENT_GRPC_VERSION
        }
    }
    generateProtoTasks {
        all().each { task ->
            task.plugins {
                javalite {}
                grpc { // Options added to --grpc_out
                    option 'lite' }
            }
        }
    }
}

The main difference is the implementation of javalite version. And I suspect that I am not able to exclude the full version correctly.

Or is there anything else I am missing ?

[ejona86]

I'd remove the protobuf 'com.google.protobuf:protobuf-java:...' and see if it works. Old versions of protobuf-lite didn't include the well-known protos, so you had to build them yourself. These days they are included in the protobuf lite jar.

[ilker-aktuna]

thanks. Tried that now. still I got the same error. what else can I try ?

[ejona86]

You don't want any.proto to be under build/extracted-protos/. Remove protobuf dependencies until you find the one that has any.proto. googleapis-common-protos doesn't have any.proto, so I feel like you have another protobuf dependency somewhere.

[ilker-aktuna]

well, thanks for that comment, but I didn't understand what to do. Do you propose to delete these 2:

    protobuf ('com.google.protobuf:protobuf-java:3.22.3') {
        exclude group: 'com.google.protobuf', module: 'protobuf-javalite'
        exclude group: 'com.google.protobuf', module: 'protobuf-java'
    }
    protobuf 'com.google.api.grpc:googleapis-common-protos:0.0.3'

I have these which might have protobuf dependency:

    implementation 'io.grpc:grpc-okhttp:1.56.1'
    implementation ('io.grpc:grpc-protobuf-lite:1.56.1') {
        exclude group: 'com.google.protobuf', module: 'protobuf-java'
    }
    implementation 'io.grpc:grpc-stub:1.56.1'

Other than that, I have proto files in the source.

Btw, I see these under extracted-protos:

[ejona86]

I thought you already tried deleting:

    protobuf ('com.google.protobuf:protobuf-java:3.22.3') {
        exclude group: 'com.google.protobuf', module: 'protobuf-javalite'
        exclude group: 'com.google.protobuf', module: 'protobuf-java'
    }

That's the one that I expect is causing the problem. I was talking about removing the protobuf types of dependencies, not implementation. If you only have those two originally (protobuf-java and googleapis-common-protos), then I don't see what could cause the problem if you removed protobuf-java and it didn't fix the problem.

[ejona86]

Btw, I see these under extracted-protos:

The extracted-protos/main/google/protobuf would have the Any (that we want to get rid of).

[ilker-aktuna]

I already removed : protobuf ('com.google.protobuf:protobuf-java:3.22.3') { exclude group: 'com.google.protobuf', module: 'protobuf-javalite' exclude group: 'com.google.protobuf', module: 'protobuf-java' }

But since it did not solve the issue, I put it back. What else can I look for ?

[ilker-aktuna]

I'd really appreciate any ideas and I'm willing to try anything suggested. I couldn't solve this issue for the last 2 months.

[ilker-aktuna]

is there a way to ignore this error and build anyway ? Because if I build "debug" version , it completes fine and runs on my device just fine. I need to update my app to comply with Google Play terms. (SDK 33) I need a quick (and maybe dirty) solution.

[ejona86]

But since it did not solve the issue, I put it back.

No, keep it removed. It is broken for newer protobuf versions. Any solution will have removing it as part of the solution.

For hacks, you might be able to do something like this:

android {
  packagingOptions {
    resources {
      pickFirsts.add("com/google/protobuf/Any$1.class")
    }
  }
}

[ilker-aktuna]

This is in app level build.gradle , right ?

When I added that , I got this error:

Build file 'C:\Development\workspace\android_studio\VectorCTRLPRO\app\build.gradle' line: 63

Could not compile build file 'C:\Development\workspace\android_studio\VectorCTRLPRO\app\build.gradle'.
> startup failed:
  build file 'C:\Development\workspace\android_studio\VectorCTRLPRO\app\build.gradle': 63: token recognition error at: '1' @ line 63, column 53.
     .add("com/google/protobuf/Any$1.class")

Then I changed the doublequotes to singlequotes. And I got the initial error as if it was not ignored:

Type com.google.protobuf.Any$1 is defined multiple times: C:\Users\ilker\.gradle\caches\transforms-3\f587bdbbf1ee885f5ee14d4cd9b8bf8b\transformed\jetified-protobuf-javalite-3.22.3.jar:com/google/protobuf/Any$1.class, C:\Development\workspace\android_studio\VectorCTRLPRO\app\build\intermediates\javac\release\classes\com\google\protobuf\Any$1.class What am I doing wrong ?

[ilker-aktuna]

I tried several different formats of this command

pickFirst pickFirsts pickFirsts.add

I tried directly inside "packagingOptions" , I also tried in "resources" within "packagingOptions" None of them worked for me.

I tried with format '**/Any$1.class' thinking that path might be wrong. But none of these worked. I'm still getting "Type com.google.protobuf.Any$1 is defined multiple times" error.

Why ?

[ilker-aktuna]

any more ideas ? Maybe you can tell me what I'm doing wrong and it does not pick the first one with pickFirst option ?

[ilker-aktuna]

please , I need help.

[ejona86]

I suggest making a reproduction. I suspect you'll find the issue while trying to make it, because the build is not reacting as I'd expect for the changes being made. But if not, you'd have a reproduction you can share and I could look more fully.

[ilker-aktuna]

What do you mean by a "reproduction" ? I already tried to compile on another PC with another Android Studio version, if that's what you're trying to say.

[ilker-aktuna]

Now, I've created a test app, simplifying it. Removed all functions and layouts. I still get the same error when building. If you want I can share this with you. Do you want me to upload it to somewhere , or send it to you ? How ?

[ejona86]

Now, I've created a test app, simplifying it.

That's what I meant with reproduction. Something you can share.

If you want I can share this with you.

Please do. Generally the easiest is to put it in a new public git repository. For example, I've made https://github.com/ejona86/grpc-java-10246 and https://github.com/ejona86/gradle-bug-map-exclude for bugs (and I've not deleted them yet because the issues aren't yet resolved). If that doesn't sound good, you can probably zip it and attach it in a comment on this issue.

[ilker-aktuna]

ok, added repo here: https://github.com/ilker-aktuna/protobuf-issue/

[ejona86]

The problem is the protobuf 'com.google.api.grpc:googleapis-common-protos:0.0.3' dependency. That is a very old artifact from 2016. You should use com.google.api.grpc:proto-google-common-protos:2.23.0 instead. But things still go wrong with that newer version.

There's two things you need: google/protobuf/descriptor.proto and google/api/annotations.proto. google/protobuf/descriptor.proto is broken in protobuf: https://github.com/protocolbuffers/protobuf/issues/7331 . The easiest workaround may be to just copy the file from protobuf to app/src/main/proto/google/protobuf/descriptor.proto. Custom message options is suspicious for an Android app, because Protobuf Lite doesn't support runtime descriptors. But if you used the options on server-side, then it'd make sense.

But something seems broken in the plugin with protobuf dependencies, as it is also trying to build protos from the transitive dependencies. I don't know why that is happening right now, but you can workaround it by excluding the transitive dependency:

    protobuf ('com.google.api.grpc:proto-google-common-protos:2.23.0') {
        exclude group: 'com.google.protobuf'
    }

With those two changes, it starts building for me.

[ilker-aktuna]

thanks. I could not understand what changes you refer to with "two changes" one is the change on com.google.api.grpc:

    protobuf ('com.google.api.grpc:proto-google-common-protos:2.23.0') {
        exclude group: 'com.google.protobuf'
    }

what is the other one ?

with only this change , I got this error:

Execution failed for task ':app:generateReleaseProto'.

protoc: stdout: . stderr: google/logging/type/http_request.proto:26:8: Option "php_namespace" unknown. Ensure that your proto definition file imports the proto which defines the option.

[ejona86]

I don't know why you'd get that error. http_request.proto is part of proto-google-common-protos. Here are the changes I made, plus creating app/src/main/proto/google/protobuf/descriptor.proto:

diff --git a/app/build.gradle b/app/build.gradle
index 334652f..40e0891 100644
--- a/app/build.gradle
+++ b/app/build.gradle
@@ -1,6 +1,6 @@
 apply plugin: 'com.android.application'
 apply plugin: 'com.google.protobuf'
-apply plugin: 'com.google.gms.google-services'
+//apply plugin: 'com.google.gms.google-services'
 //apply plugin: 'io.fabric'

 android {
@@ -82,7 +82,9 @@ dependencies {
     implementation 'javax.annotation:javax.annotation-api:1.2'
     //implementation 'com.google.protobuf:protobuf-lite:3.0.0'

-    protobuf 'com.google.api.grpc:googleapis-common-protos:0.0.3'
+    protobuf ('com.google.api.grpc:proto-google-common-protos:2.23.0') {
+        exclude group: 'com.google.protobuf'
+    }
     //implementation 'io.netty:netty-tcnative-parent:2.0.20.Final'
     //implementation 'io.netty:netty:4.0.0.Alpha8'
     //implementation 'com.google.android.gms:play-services:12.0.1'
@@ -100,6 +102,7 @@ dependencies {
     implementation 'com.google.android.play:core:1.10.3'
     implementation 'com.google.guava:guava:28.2-android'
     implementation project(path: ':nativetemplates')
+    testImplementation 'junit:junit:4.13.2'

     constraints {
         implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.0") {

[ilker-aktuna]

what do you mean by "creating app/src/main/proto/google/protobuf/descriptor.proto" ? I didn't do that ?

[ejona86]

what do you mean by "creating app/src/main/proto/google/protobuf/descriptor.proto" ?

I was referencing the part I said earlier:

The easiest workaround may be to just copy the file from protobuf to app/src/main/proto/google/protobuf/descriptor.proto.

This file.

[ilker-aktuna]

OK. I put the descriptor.proto to that file. But now I get this error/warning:

Execution failed for task ':app:generateReleaseProto'.

protoc: stdout: . stderr: C:\Development\workspace\android_studio\0Denemeler\Testapp\app\build\extracted-protos\main\google\protobuf\descriptor.proto: Input is shadowed in the --proto_path by "C:/Development/workspace/android_studio/0Denemeler/Testapp/app/src/main/proto/google/protobuf/descriptor.proto". Either use the latter file as your input or reorder the --proto_path so that the former file's location comes first.

How can I do one of these: use the latter file as your input or reorder the --proto_path so that the former file's location comes first.

[ejona86]

The exclude group: 'com.google.protobuf' is what should have avoided that error. I don't know what to tell you other than your build worked on my machine with those changes I made. When I built, there was no descriptor.proto in extracted-proto/. You might try ./gradlew clean and building again.

[ilker-aktuna]

ok. after a clean it built successfully. Now I'll try this solution on the real app and see if the proto functions work fine.

[ilker-aktuna]

The code compiled and the proto functions are working fine. Thanks for all your help @ejona86 for continuing to help tirelessly despite all my inexperience.

[ejona86]

I've created #735 to fix the need for doing gradle clean.

[ilker-aktuna]

will this fix remove the need for adding the proto file manually ?

[ejona86]

The fix only makes sure to delete now-gone protos. So if you add a dependency, do a build, remove that dependency, and do a build you won't see the generated code for that now-gone dependency.