protoc-gen-grpc-java not available on apple m1

[reinmind]

What version of gRPC-Java are you using?

1.34.0

What is your environment?

osx 11.0.1

What did you expect to see?

build success

What did you see instead?

[INFO] --- protobuf-maven-plugin:0.6.1:compile (default-cli) @ grpc ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.720 s
[INFO] Finished at: 2020-12-03T16:37:02+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.6.1:compile (default-cli) on project grpc: Unable to resolve artifact: Missing:
[ERROR] ----------
[ERROR] 1) com.google.protobuf:protoc:exe:osx-aarch_64:3.12.0
[ERROR] 
[ERROR]   Try downloading the file manually from the project website.
[ERROR] 
[ERROR]   Then, install it using the command: 
[ERROR]       mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.12.0 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR] 
[ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
[ERROR]       mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.12.0 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR]   Path to dependency: 
[ERROR]     1) com.reinmind:grpc:jar:1.0-SNAPSHOT
[ERROR]     2) com.google.protobuf:protoc:exe:osx-aarch_64:3.12.0
[ERROR] 
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR] 
[ERROR] for artifact: 
[ERROR]   com.reinmind:grpc:jar:1.0-SNAPSHOT
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR]   aliyunmaven (https://maven.aliyun.com/repository/public, releases=true, snapshots=false)
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Steps to reproduce the bug

mvn protobuf:compile

[ejona86]

Yes, we do not produce binaries for arm64 macs.

---

Edit: Summary of the long issue: The easiest thing to do as a consumer is to install Rosetta. With it installed, the Intel binary works fine. If you'd like to contribute, see https://github.com/grpc/grpc-java/issues/7690#issuecomment-831617279 . Easiest plan is to build protobuf and the grpc plugin as universal binaries.

[ejona86]

Since this is the grpc repo, I've called out protoc-gen-grpc-java not being available. Although your actual error is protoc itself, which would be part of the protobuf project. If grpc gets to building it first, then we'd need to add support in protobuf as well.

[gaurav46]

Hi @ejona86 Is it possible for any workaround like to set osdetector.arch as x86_64 to get the complication done ? I tried to set below in gradle.properties but it still looks for osx-aarch_64 binaries from maven. org.gradle.jvmargs=-Dos.detected.name=osx -Dos.detected.arch=x86_64 -Dos.detected.classifier=osx-x86_64

[ejona86]

To manually specify the classifier, for Gradle:

protobuf {
  protoc {
    artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
  }
}

I'm less familiar with the Maven plugin, but I think it may be something like:

<protocArtifact>com.google.protobuf:protoc:3.14.0:exe:osx-x86_64</protocArtifact>

You can do the same for protoc-gen-grpc-java.

[gaurav46]

Thanks @ejona86 . That worked out.

[zjc17]

Thanks @ejona86 , But How can I keep other platform like linux-x86_64 when the project being compiled on other platform?

[ejona86]

@Jiachen-Zhang

protobuf {
  protoc {
    if (osdetector.os == "osx") {
      artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
    } else {
      artifact = 'com.google.protobuf:protoc:3.14.0'
    }
  }
}

You can also use project properties to only change the behavior when explicitly requested.

[zjc17]

@Jiachen-Zhang

protobuf {
  protoc {
    if (osdetector.os == "osx") {
      artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
    } else {
      artifact = 'com.google.protobuf:protoc:3.14.0'
    }
  }
}

You can also use project properties to only change the behavior when explicitly requested.

Thanks!

Finally, I made it by

// for apple m1, please add protoc_platform=osx-x86_64 in $HOME/.gradle/gradle.properties
if (project.hasProperty('protoc_platform')) {
    artifact = "com.google.protobuf:protoc:3.13.0:${protoc_platform}"
} else {
    artifact = "com.google.protobuf:protoc:3.13.0"
}

[hu-chia]

for me, add this to ~/.m2/settings.xml, it worked:

<settings>
  ...
  <activeProfiles>
    <activeProfile>
      apple-silicon
    </activeProfile>
    ...
  </activeProfiles>
  <profiles>
    <profile>
      <id>apple-silicon</id>
      <properties>
        <os.detected.classifier>osx-x86_64</os.detected.classifier>
      </properties>
    </profile>
    ...
  </profiles>
  ...
</settings>

[Uditmittal]

Can you share the full setting.xml I tried it didn't work for me? I am using M1 with protobuf.

[zjc17]

@Uditmittal in my build.gradle file

protobuf {
    protoc {
        // for apple m1, please add protoc_platform=osx-x86_64 in $HOME/.gradle/gradle.properties
        if (project.hasProperty('protoc_platform')) {
            artifact = "com.google.protobuf:protoc:3.13.0:${protoc_platform}"
        } else {
            artifact = "com.google.protobuf:protoc:3.13.0"
        }
    }
}

in my ~/.gradle/gradle.properties file

protoc_platform=osx-x86_64

I think you can define and use variables in maven to implement this feature. Good luck

[hu-chia]

@Uditmittal My settings.xml is awfully long, the key is using your property overwrite kr.motd.maven:os-maven-plugin detected os classifier, if you share your settings.xml and pom.xml snippet, this may be help.

[Uditmittal]

<settings>
  <activeProfiles>
    <activeProfile>
      apple-silicon
    </activeProfile>
  </activeProfiles>
  <profiles>
    <profile>
      <id>apple-silicon</id>
      <properties>
        <os.detected.classifier>osx-x86_64</os.detected.classifier>
      </properties>
    </profile>
  </profiles>
</settings>

@guyeu my seetings.xml file

[hu-chia]

@Uditmittal This is my org.xolstice.maven.plugins:protobuf-maven-plugin build plugin configuration, the ${os.detected.classifier} value could read from active profile from settings.xml.

      <configuration>
          <protocArtifact>
              com.google.protobuf:protoc:3.1.0:exe:${os.detected.classifier}
          </protocArtifact>
          <pluginId>grpc-java</pluginId>
          <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.0.0:exe:${os.detected.classifier}</pluginArtifact>
      </configuration>

[Uditmittal]

Thanks, it works for me.

[bangbang2333]

谢谢老哥，搞定了

[ejona86]

I'm not at all impressed with how hard it was to find the appropriate flags to pass to clang to make a universal binary, but it appears to be -arch arm64 -arch x86_64. Unclear if cross-compiling is as simple as -arch arm64. We can start by figuring out how to cross-compile/universal binary for protobuf. Afterward we can fight Gradle. The goal would be to let us build M1 binaries from an x86 Mac. But I don't have a Mac which means I can't help much, as develop-via-CI is a form of torture.

[ghost]

@ejona86 Have you had any success building from source for AArch64? I am interested in helping moving this along...

[ejona86]

@jjones-figure, no, as I said, I don't have a Mac, so I can't really help much here. We totally build from source for AArch64 already on Linux. But it is a different toolchain for M1.

[ejona86]

https://github.com/protocolbuffers/protobuf/pull/8557 just went into protobuf based on my suggestion to just copy the current x86 binary to an arm64 name until we have actual M1 support. That doesn't "fix" this, but it does resolve the main user-visible issues and doesn't turn out to be too hacky.

I think that is as simple as making a copy of the "exe" and the hashes/signature in our upload_artifacts.sh script.

[kdubb]

@ejona86 Where do I beg to get protoc-gen-grpc-java published as osx-aarch_64?

This is the lone holdout of 3 projects (including com.google.protobuf:protoc) required for building easily on the M1. The other projects seem to be following the advice of copying osx-x86_64 to osx-aarch_64 and it works well. I currently do this manually on my machine to get the required protoc-gen-grpc-java:osx-aarch_64 artifact.

[kdubb]

@ejona86 I've had a look at the repo and created a very simple PR #8680 . No idea if it works but maybe this gets things rolling?

[liuxuliangcumt]

@Uditmittal in my build.gradle file

protobuf {
    protoc {
        // for apple m1, please add protoc_platform=osx-x86_64 in $HOME/.gradle/gradle.properties
        if (project.hasProperty('protoc_platform')) {
            artifact = "com.google.protobuf:protoc:3.13.0:${protoc_platform}"
        } else {
            artifact = "com.google.protobuf:protoc:3.13.0"
        }
    }
}

in my ~/.gradle/gradle.properties file

protoc_platform=osx-x86_64

I think you can define and use variables in maven to implement this feature. Good luck

in flutter_blue android. build.gradle add protobuf... you say in my build.gradle file .how could we know in' flutter_blue android build.gradle 'to add ?

[solohsu]

https://gist.github.com/solohsu/a2bac0498ef51c105cedd2dc0a854bcf I write a more general Gradle workaround script which may be helpful to you guys who come across any plugin that doesn't support macOS M1 Soc, not only protobuf.

[liuxuliangcumt]

谢谢您的来信！

[ilya-yarets]

To manually specify the classifier, for Gradle:

protobuf {
  protoc {
    artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
  }
}

I'm less familiar with the Maven plugin, but I think it may be something like:

<protocArtifact>com.google.protobuf:protoc:3.14.0:exe:osx-x86_64</protocArtifact>

You can do the same for protoc-gen-grpc-java.

thank you very much! it's working

[ejona86]

Reopening because we've only a workaround currently. We still need to do something along the lines of https://github.com/grpc/grpc-java/issues/7690#issuecomment-831617279 for native M1 support. I thought we had another issue open for tracking native M1 support, but I can't find it. Anyway, this will do.

[liuxuliangcumt]

谢谢您的来信！

[sukhbir-singh]

Is this solved ? I am still facing this issue.

[ejona86]

Interactions with the protobuf plugins is resolved, but requires Rosetta. I reopened the issue to track native M1 support.

[coding-devops]

To manually specify the classifier, for Gradle:

protobuf {
  protoc {
    artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
  }
}

I'm less familiar with the Maven plugin, but I think it may be something like:

<protocArtifact>com.google.protobuf:protoc:3.14.0:exe:osx-x86_64</protocArtifact>

You can do the same for protoc-gen-grpc-java.

why it's not work for me .... >_<

[liuxuliangcumt]

谢谢您的来信！

[mflis]

@ejona86 Is adding native M1 support dependent on resolving this issue first?

I have Macbook M1 available, so I could experiment a bit, but I’m not very familiar with this repository, so I would appreciate pointing me in the right direction.

[ejona86]

@mflis, it is independent but related. If either us (grpc-wide, not just java) or protobuf confirm how to crosscompile, then it will probably make the process a bit faster for the others. M1 macs are becoming a bit more common (the likelihood you know someone that has one), which is making this easier to test as time goes on.

[hakusai22]

[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.0:compile (default) on project lollypop-proto: An error occurred while invoking protoc.: Error while executing process. Cannot run program "/Users/yinpeng/JavaWorkSpace/lollypop/lollypop-server/lollypop-proto/target/protoc-plugins/protoc-3.1.0-osx-x86_64.exe": error=86, Bad CPU type in executable -> [Help 1] [ERROR] @guyeu maven 换完之后执行 还是报错

[ejona86]

@xizizzz, you need Rosetta. That is expected. See https://github.com/google/protobuf-gradle-plugin/issues/543 if you want to understand why you get that failure.

[amihaiemil]

For maven users, add this profile:

<profile>
    <id>m1</id>
    <build>
        <plugins>
            <plugin>
                <groupId>com.github.os72</groupId>
                 <artifactId>protoc-jar-maven-plugin</artifactId>
                 <executions>
                     <execution>
                         <phase>generate-sources</phase>
                         <goals>
                             <goal>run</goal>
                         </goals>
                         <configuration>
                             <protocArtifact>com.google.protobuf:protoc:3.1.0:exe:osx-x86_64</protocArtifact>
                         </configuration>
                     </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>

The important thing is:

com.google.protobuf:protoc:3.1.0:exe:``osx-x86_64``

With osx-x86_64, the plugin knows to use the exe for Apple M1.

Then just mvn clean install -Pm1 should work.

[ejona86]

@amihaiemil, protoc:3.1.0 is ancient. Users should upgrade to a newer version of protoc that has a (fake; copy of x86_64) osx-aarch_64 artifact. In either case you'd need Rosetta.

[amihaiemil]

@ejona86 Thanks, but what do you mean with (fake; copy of x86_64)?

[ejona86]

@amihaiemil, see https://github.com/protocolbuffers/protobuf/pull/8557

[MochiXu]

@ejona86 Would you please tell me what's your Gradle version, my Gradle still try to download the aarch64 version after using your suggestion

[ejona86]

@MochiXu, which of my suggestions? And what error are you seeing?

[liuxuliangcumt]

谢谢您的来信！

[MochiXu]

@MochiXu, which of my suggestions? And what error are you seeing?

Thanks for your reply! I see your suggestions from this link: https://github.com/grpc/grpc-java/issues/7690#issuecomment-747196215)，my computer use m1 chip, so when I try to use Gradle to build protobuf, Gradle will try to download protoc-3.14.0-osx-arrch_64.exe from https://repo1.maven.org/maven2/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-arrch_64.exe, but this repository doesn't have protoc-3.14.0-osx-arrch_64.exe. So under your suggestion, I changed artifact = 'com.google.protobuf:protoc:3.14.0' with artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64', but it doesn't work, it is still downloading form https://repo1.maven.org/maven2/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-arrch_64.exe

[ejona86]

@MochiXu, I expect you're just missing a place in your build that you need to make the change. There's not much way to help you without seeing your build. That said, that recommendation of mine is pretty old. Since protobuf v3.17.3, protobuf has been copying the osx-x86_64 artifact to osx-aarch_64 meaning you wouldn't need configuration in your build. So these days I'd recommend you just upgrade to a newer protobuf.

[MochiXu]

@MochiXu, I expect you're just missing a place in your build that you need to make the change. There's not much way to help you without seeing your build. That said, that recommendation of mine is pretty old. Since protobuf v3.17.3, protobuf has been copying the osx-x86_64 artifact to osx-aarch_64 meaning you wouldn't need configuration in your build. So these days I'd recommend you just upgrade to a newer protobuf.

It works!👍 Thanks for your kindness🙏, I changed my protobuf to v3.17.3 and grpc to v1.45.1✌️

[wallezhang]

@ejona86 Hi, I found the protoc-gen-grpc-java program is still not available on apple M1 chip. It seems the copy of x86 version, so I received an error message: program not found or is not executable when I compile the proto files.

[chenzhiguo]

Bump grpc-java.version to 1.47.0 and protobuf-java.version to 3.21.1 for Apple Silicon is ok.

[wallezhang]

Bump grpc-java.version to 1.47.0 and protobuf-java.version to 3.21.1 for Apple Silicon is ok.

Actually protoc-gen-grpc-java is not compatible with m1 chip since bump grpc-java version to 1.47.0. protoc-gen-grpc-java-1.47.0-osx-aarch_64.exe and protoc-gen-grpc-java-1.47.0-osx-x86_64.exe has the same md5, so there is no native compatible with m1 chip.

If you don't install Rosetta on mac, then you will receive an error when execute protoc-gen-grpc-java-1.47.0-osx-aarch_64.exe. And protobuf-java is compatible with m1 chip natively.

[Jalaja-Apexon]

Hi, I am facing this issue while building my grpc project in Java using Maven build

[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.9.2:run (default) on project grpc-Oscar-Health: Error resolving artifact: com.google.protobuf:protobuf-java:jar:3.9.2:exe:osx-x86_64: Failure to find com.google.protobuf:protobuf-java:3.9.2:exe:jar in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
[ERROR] 
[ERROR] Try downloading the file manually from the project website.
[ERROR] 
[ERROR] Then, install it using the command: 
[ERROR]     mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protobuf-java -Dversion=jar -Dclassifier=exe -Dpackaging=3.9.2 -Dfile=/path/to/file
[ERROR] 
[ERROR] Alternatively, if you host your own repository you can deploy the file there: 
[ERROR]     mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protobuf-java -Dversion=jar -Dclassifier=exe -Dpackaging=3.9.2 -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR] 
[ERROR]   com.google.protobuf:protobuf-java:3.9.2:jar
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR]   central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

My pom.xml file is as follows: -

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>grpc-Oscar-Health</groupId>
  <artifactId>grpc-Oscar-Health</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>grpc-Oscar-Health</name>
  <dependencies>
    <dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>3.9.2</version>
        </dependency>

        <dependency>
             <groupId>io.grpc</groupId>
            <artifactId>grpc-all</artifactId>
             <version>1.47.0</version>
        </dependency>

        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-netty-shaded</artifactId>
            <version>1.22.1</version>
        </dependency>
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-protobuf</artifactId>
            <version>1.22.1</version>
        </dependency>
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-stub</artifactId>
            <version>1.22.1</version>
        </dependency>

        <dependency>
            <groupId>javax.annotation</groupId>
            <artifactId>javax.annotation-api</artifactId>
            <version>1.3.2</version>
        </dependency>

        <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-simple</artifactId>
        <version>1.7.28</version>
    </dependency>

    </dependencies>

        <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
    </properties>

    <build>
        <defaultGoal>clean generate-sources compile install</defaultGoal>

        <plugins>
            <!-- compile proto file into java files. -->
            <plugin>
                <groupId>com.github.os72</groupId>
                <artifactId>protoc-jar-maven-plugin</artifactId>
                <version>3.9.2</version>
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                <configuration>
                    <protocVersion>3.9.2</protocVersion>
                    <includeStdTypes>true</includeStdTypes>
                    <protocArtifact>com.google.protobuf:protobuf-java:jar:3.9.2:exe:osx-x86_64</protocArtifact>
                    <pluginArtifact>io.grpc:protoc-gen-grpc-java:pom:1.47.0</pluginArtifact>
                            <includeMavenTypes>direct</includeMavenTypes>

                            <includeMavenTypes>transitive</includeMavenTypes>

                            <addProtoSources>all</addProtoSources>

                            <includeDirectories>
                                <include>src/main/resources</include>
                            </includeDirectories>
                                <inputDirectories>
                                <include>src/main/resources/protobuf/src</include>
                            </inputDirectories>
                            <inputDirectories>
                                <include>src/main/resources/protobuf/src/hioscar</include>
                            </inputDirectories>                                                     
                            <outputTargets>
                                <outputTarget>
                                    <type>java</type>
                                    <outputDirectory>src/main/java</outputDirectory>
                                </outputTarget>
                            </outputTargets>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Kindly suggest suitable resolution for the same. The entire setup is in Mac device.