missing pre-compiled arm binary

[shotor]

Problem description

Can't install grpc-tools on arm architecture (raspberry pi) because of missing precompiled binaries.

node-pre-gyp ERR! stack Error: 404 status code downloading tarball https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz

Is it possible to build these at all?

Reproduction steps

npm init npm install grpc-tools

Environment

• Ubuntu 20.04 arm64
  • 10-12-14
  • n
  • grpc-tools, possibly more grpc packages - haven't checked yet

Additional context

error /home/repo/grpc/node_modules/grpc-tools: Command failed.
Exit code: 1
Command: node-pre-gyp install
Arguments:
Directory: /home/repo/grpc/node_modules/grpc-tools
Output:
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using node-pre-gyp@0.12.0
node-pre-gyp info using node@10.20.1 | linux | arm64
node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp info check checked for "/home/repo/grpc/node_modules/grpc-tools/bin/grpc_tools.node" (not found)
node-pre-gyp http GET https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
node-pre-gyp http 404 https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
node-pre-gyp ERR! install error
node-pre-gyp ERR! stack Error: 404 status code downloading tarball https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
node-pre-gyp ERR! stack     at Request.<anonymous> (/home/repo/grpc/node_modules/node-pre-gyp/lib/install.js:149:27)
node-pre-gyp ERR! stack     at Request.emit (events.js:203:15)
node-pre-gyp ERR! stack     at Request.onRequestResponse (/home/repo/grpc/node_modules/request/request.js:1059:10)
node-pre-gyp ERR! stack     at ClientRequest.emit (events.js:198:13)
node-pre-gyp ERR! stack     at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:565:21)
node-pre-gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:111:17)
node-pre-gyp ERR! stack     at TLSSocket.socketOnData (_http_client.js:451:20)
node-pre-gyp ERR! stack     at TLSSocket.emit (events.js:198:13)
node-pre-gyp ERR! stack     at addChunk (_stream_readable.js:288:12)
node-pre-gyp ERR! stack     at readableAddChunk (_stream_readable.js:269:11)
node-pre-gyp ERR! System Linux 5.4.0-1008-raspi
node-pre-gyp ERR! command "/usr/local/bin/node" "/home/repo/grpc/node_modules/grpc-tools/node_modules/.bin/node-pre-gyp" "install"
node-pre-gyp ERR! cwd /home/repo/grpc/node_modules/grpc-tools
node-pre-gyp ERR! node -v v10.20.1
node-pre-gyp ERR! node-pre-gyp -v v0.12.0
node-pre-gyp ERR! not ok

[Nilos]

This is also an issue when running inside Docker on the new MacBook with the M1 Chip.

[antonimmo]

@Nilos Same issue brought me here 👋

Update: I managed to build and run the container by adding --platform linux/amd64 option to both build and run docker commands. You could alternatively specify platform: linux/amd64 for each docker-compose service.

[remko]

Given the introduction of Apple Silicon, and the fact that Docker for Mac on these machines use arm64, does it make sense to re-open this case? Building JS apps that depend on grpc (directly or indirectly) is no longer possible in Docker on Mac (or at least not easy; e.g I can't make the platform trick work with docker-compose, and there are many qemu crashes when running in emulated mode on an M1).

[antonimmo]

@remko is right, I'd reopen this one. Some issues I came across with just can't be sorted out that easily.

[murgatroid99]

Our build process runs on GitHub Actions, so adding this is blocked on actions/virtual-environments#2187.

[sri-vathsa]

Can anyone share the instructions on how to build the binaries from the source code?

[nicolasnoble]

Can anyone share the instructions on how to build the binaries from the source code?

Basically https://github.com/grpc/grpc-node/blob/master/packages/grpc-tools/build_binaries.sh

This script probably requires changes, but we have no way of testing on any M1 hardware, so you'd be on your own to change it.

[sri-vathsa]

@nicolasnoble @murgatroid99 I am actually using trying to install the grpc-tools on AWS Graviton 2 instances. Are there any instructions on how to run the test cases? I have access to both M1 hardware and Graviton instances. I can run the test cases and share my findings/results.

[shnhrrsn]

For any on M1 hardware running into this, I was able to workaround it by just forcing the x64 binary and letting Rosetta deal with it:

yarn add grpc-tools --ignore-scripts
pushd node_modules/grpc-tools
node_modules/.bin/node-pre-gyp install --target_arch=x64
popd

Is there any way to include this during normal install?

[murgatroid99]

I think you can pass the --target_arch=x64 argument directly to npm install. I assume it works the same way with yarn, but I don't use yarn.

[antonimmo]

@shnhrrsn Also, if you use nvm, you can install x86 node versions following the steps under the "Macs with M1 chip" section of https://github.com/nvm-sh/nvm#macos-troubleshooting. This one solved all my M1 related issues.

[circa10a]

ARM binaries should be distributed as part of releases. Especially in 2021. Many cloud providers and personal computer manufacturers are beginning to implement ARM as larger offerings. This is also troublesome for raspberry pi users who would like to run node apps. There isn't a good reason not to pre-compile these binaries, otherwise node marketshare will lessen. I'm already pretty frustrated with it and will be doing less node development as a result.

[circa10a]

@murgatroid99 Any chance this group can reconsider?

[murgatroid99]

I said that making this change is blocked in https://github.com/grpc/grpc-node/issues/1405#issuecomment-784609436. Nothing in the linked issue appears to have changed, so it's still blocked.

[artursapek]

Our build process runs on GitHub Actions, so adding this is blocked on actions/virtual-environments#2187.

lol that issue is blocking SO many people

[ryanberckmans]

We do not intend to distribute ARM binaries for grpc-tools. I suggest generating your code on a different system and then transferring your code to your ARM system instead.

Apple Silicon is here to stay and so is grpc-node. Please reconsider :)

[murgatroid99]

That comment predates the announcement of Apple Silicon. I wasn't talking about Apple Silicon. As I said in my more recent comments https://github.com/grpc/grpc-node/issues/1405#issuecomment-784609436 and https://github.com/grpc/grpc-node/issues/1405#issuecomment-891187791

Our build process runs on GitHub Actions, so adding this is blocked on actions/virtual-environments#2187.

[sebirdman]

I'm curious why there's not a fallback setting to just compile the binaries if no prebuilt exists. Other packages do something similar.

Edit: it appears node-pre-gyp is being abused to just download files. because we're not actually using a binding to node for those tools?

[murgatroid99]

I disagree with the term "abused". We are using node-pre-gyp to do what it is supposed to do: download binary files for the system the code is running on. It doesn't have a fallback to building locally because it is building protoc, which has an existing build process that uses cmake instead of node-gyp.

[sebirdman]

Abused might be a strong word, but it's certainly a tad misused. Their docs say it should "stand between" npm and node-gyp. If we can't fallback using the node-gyp to build, i wonder if there's a good way to wrap node-pre-gyp to just auto run ./build_binaries.sh if that fails?

Here's how i got around this without using Rosetta:

1) clone this repo 2) install your grpc-tools with yarn install --ignore-scripts 3) run the script ./build_binaries.sh in the grpc-tools package 4) copy your files over, something like this:

#!/bin/bash

cp binaries/arm64/protoc node_modules/grpc-tools/bin/protoc
cp binaries/arm64/grpc_node_plugin node_modules/grpc-tools/bin/grpc_node_plugin

[nicolasnoble]

Think of it more as a stop gap, really. The reality being that building protoc from pure source is impossible due to its bootstrapping nature, and so we can't write a node-gyp installer for it, which requires a very flat list of files to compile, with no intermediate compilation process.

Ideally, node-gyp should support cmake, but that's also not happening: https://github.com/nodejs/node-gyp/issues/1590

Additionally, Apple has made it very difficult to build software on their new M1 hardware when using things like github actions, rendering us basically powerless to create working M1 binaries. So we're stuck between a rock and a hard place here, and there's no good solution unfortunately.

[NixBiks]

This is also an issue when running inside Docker on the new MacBook with the M1 Chip.

Did anyone find a fix when running in Docker?

We have a team that develops in VSCode Remote Containers (ie. docker containers) - some have M1 chips others intel. Any way to support both?

[NixBiks]

@shnhrrsn Also, if you use nvm, you can install x86 node versions following the steps under the "Macs with M1 chip" section of https://github.com/nvm-sh/nvm#macos-troubleshooting. This one solved all my M1 related issues.

@antonimmo That doesn't work when inside a debian-bullseye docker container (on a Mac M1 host) since arch -x86_64 zsh causes an error.

[kakyoism]

Looks like we must build grpc-tools ourselves for Apple Silicon? I'm a beginner here. Could anybody kindly share a pointer to how to build it locally?

[ldx]

Looks like we must build grpc-tools ourselves for Apple Silicon? I'm a beginner here. Could anybody kindly share a pointer to how to build it locally?

Clone this repo, then cd grpc-node/packages/grpc-tools; git submodule update --init --recursive; ./build_binaries.sh should build the binaries. Make sure you have XCode/Command Line Tools installed.

[kakyoism]

@ldx Thanks a lot! I also found that I could resort to the emulated version with a Rosseta-enabled Terminal copy and install the Intel-edition Homebrew.

[NixBiks]

Kind of off-topic question for the workaround to work.

Is it possible to list the built binary in package.json as fallback when a binary is not available? My issue is that grpc-tools are listed as a dependency which works for all developers except those with M1.

[vitoorgomes]

@ldx Thanks a lot! I also found that I could resort to the emulated version with a Rosseta-enabled Terminal copy and install the Intel-edition Homebrew.

could you please give more details on how you did all of that?

[ldx]

Kind of off-topic question for the workaround to work.

Is it possible to list the built binary in package.json as fallback when a binary is not available? My issue is that grpc-tools are listed as a dependency which works for all developers except those with M1. For M1 I want to use the compiled darwin-x64.tar.gz instead.

It is possible and would be nice to be able to build it if a precompiled binary is not available.

[davetisyan95]

I am on an apple m1 chip.

My workaround.....

arch -x86_64 bash
touch ~/.bashrc
curl -sL https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.0/install.sh -o install_nvm.sh

# I wanted to use node 14, so I had to uninstall and reinstall with new arch
nvm uninstall 14
nvm install 14

# navigate to your project dir in the same terminal window running diff arch
cd ~/<my-project-dir>
rm -rf node_modules && rm package-lock.json
npm i

# run start command
npm start

[ldx]

If someone wants to do this in a less hackish way: build the binaries for ARM64, then create a tarball named darwin-arm64.tar.gz with these two files in it:

drwxr-xr-x root/root         0 2021-12-07 12:13 bin/
-rwxr-xr-x root/root  12023196 2021-12-07 12:13 bin/protoc
-rwxr-xr-x root/root   5945229 2021-12-07 12:13 bin/grpc_node_plugin

You can use an S3 bucket as a mirror, upload the tarball under the path grpc-tools/v1.11.2/darwin-arm64.tar.gz (match the version to whatever version of grpc-tools you use), make it public, and then npm install --grpc_tools_binary_host_mirror=https://<my-bucket>.s3.amazonaws.com/ should work. You might also want to mirror the tarballs for other platforms, e.g. https://node-precompiled-binaries.grpc.io/grpc-tools/v1.11.2/darwin-x64.tar.gz and https://node-precompiled-binaries.grpc.io/grpc-tools/v1.11.2/linux-x64.tar.gz if you also use x86-64 Mac OS or Linux.

[NixBiks]

Did anyone manage to install grpc-tools inside a vscode devcontainer on a M1 mac?

I can install grpc-tools by using the solution suggested by @davetisyan95 but I can't do that inside a docker container.

[NixBiks]

I'd also love to see a full example with the suggestion from @ldx - Basically I have grpc-tools as dependencies in package.json and I want to be able to run yarn install on a M1 and non-M1 machine (inside docker containers). I haven't figured out how to do that

[fredrikaverpil]

Did anyone manage to install grpc-tools inside a vscode devcontainer on a M1 mac?

I can install grpc-tools by using the solution suggested by @davetisyan95 but I can't do that inside a docker container.

Did you try what was mentioned here?

[NixBiks]

Did anyone manage to install grpc-tools inside a vscode devcontainer on a M1 mac? I can install grpc-tools by using the solution suggested by @davetisyan95 but I can't do that inside a docker container.

Did you try what was mentioned here?

Yeah I use docker compose up and then I get

Error response from daemon: image with reference node:16-bullseye was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64/v8

[NixBiks]

Actually I just realized that I should first delete the existing image with arm64 architecture - I'll report back

Update

It might solve it - now I'm just getting different errors and I can't say if this is related to this or not yet

[NixBiks]

If someone wants to do this in a less hackish way: build the binaries for ARM64, then create a tarball named darwin-arm64.tar.gz with these two files in it:

drwxr-xr-x root/root         0 2021-12-07 12:13 bin/
-rwxr-xr-x root/root  12023196 2021-12-07 12:13 bin/protoc
-rwxr-xr-x root/root   5945229 2021-12-07 12:13 bin/grpc_node_plugin

You can use an S3 bucket as a mirror, upload the tarball under the path grpc-tools/v1.11.2/darwin-arm64.tar.gz (match the version to whatever version of grpc-tools you use), make it public, and then npm install --grpc_tools_binary_host_mirror=https://<my-bucket>.s3.amazonaws.com/ should work. You might also want to mirror the tarballs for other platforms, e.g. https://node-precompiled-binaries.grpc.io/grpc-tools/v1.11.2/darwin-x64.tar.gz and https://node-precompiled-binaries.grpc.io/grpc-tools/v1.11.2/linux-x64.tar.gz if you also use x86-64 Mac OS or Linux.

Did anyone succeed with this approach? I.e. making it possible to simply yarn yarn install and then it'll install grpc-tools regardlass of architecture.

[NixBiks]

@ldx I've got it working with npm install --grpc_tools_binary_host_mirror=https://<my-bucket>.s3.amazonaws.com/. Thanks for that tip.

Does anyone know the yarn equivalent (yarn install --grpc_tools_binary_host_mirror=https://<my-bucket>.s3.amazonaws.com/ does not work)? And is it possible to set some config so yarn install automatically will use the mirror if the package is not found in the official registry?

Can anyone explain where --grpc_tools_binary_host_mirror actually comes from? I can't find any documentation on it.

Update

Just found this issue - unfortunately no answer so not sure if it's possible with yarn.

[daimatz]

Clone this repo, then cd grpc-node/packages/grpc-tools; git submodule update --init --recursive; ./build_binaries.sh should build the binaries. Make sure you have XCode/Command Line Tools installed.

@ldx Thank you, I've made it work with your solution. But in my environment (Oracle Cloud Ampere A1 Compute, Ubuntu-20.04) I needed to remove -m32 arguments from CMAKE_CXX_FLAGS, CMAKE_C_FLAGS, CMAKE_EXE_LINKER_FLAGS.

The following changes worked:

$ git diff packages/grpc-tools/linux_32bit.toolchain.cmake
diff --git a/packages/grpc-tools/linux_32bit.toolchain.cmake b/packages/grpc-tools/linux_32bit.toolchain.cmake
index 5a1b80f0..62347c7f 100644
--- a/packages/grpc-tools/linux_32bit.toolchain.cmake
+++ b/packages/grpc-tools/linux_32bit.toolchain.cmake
@@ -1,3 +1,3 @@
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -m32" CACHE STRING "c++ flags")
-set(CMAKE_C_FLAGS   "${CMAKE_C_FLAGS} -m32" CACHE STRING "c flags")
-set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -m32" CACHE STRING "ld flags")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}" CACHE STRING "c++ flags")
+set(CMAKE_C_FLAGS   "${CMAKE_C_FLAGS}" CACHE STRING "c flags")
+set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS}" CACHE STRING "ld flags")

My environment:

$ uname -a
Linux instance-20220101-1510 5.11.0-1022-oracle #23~20.04.1-Ubuntu SMP Fri Nov 12 15:45:47 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

[maschwenk]

I was able to build it on an Apple Silicon Mac like so:

git clone git@github.com:grpc/grpc-node.git
cd grpc-node
brew install cmake # ./build_binaries.sh needs cmake. also tried xcode-select --install but it didn't seem to work
cd packages/grpc-tools && git submodule update --init --recursive && ./build_binaries.sh

# then to install in pnpm (does not support npm flag)
npm_config_grpc_tools_binary_host_mirror="https://github.com/maschwenk/grpc-node/raw/2dd28e1ab8211533007dd2df5ae632de60006983/artifacts/" pnpm install
# checkout https://github.com/mapbox/node-pre-gyp/blob/master/lib/util/versioning.js#L316
# seems passing as an ENV also works (so will probably work for PNPM and Yarn)

See more here https://github.com/grpc/grpc-node/compare/maschwenk:grpc-tools%401.11.2-apple-silicon-build#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R50

You will also need to manually change packages/grpc-tools/build_binaries.sh to not set the arch to x64.

Can anyone explain where --grpc_tools_binary_host_mirror actually comes from? I can't find any documentation on it.

@mr-bjerre you can pass it through the env

npm_config_grpc_tools_binary_host_mirror="https://github.com/maschwenk/grpc-node/raw/2dd28e1ab8211533007dd2df5ae632de60006983/artifacts/" yarn install

Note that it doesn't link directly to the file, but the folder containing the binaries

[srabraham]

@mr-bjerre you can pass it through the env

npm_config_grpc_tools_binary_host_mirror="https://github.com/maschwenk/grpc-node/raw/2dd28e1ab8211533007dd2df5ae632de60006983/artifacts/" yarn install

Note that it doesn't link directly to the file, but the folder containing the binaries

@maschwenk is my hero. I was subsequently, finally, able to get my Bazel workspace to build, simply by adding an environment bit to my WORKSPACE's yarn_install

yarn_install(
    name = "npm",
    package_json = "//:package.json",
    yarn_lock = "//:yarn.lock",
    environment = {
        "npm_config_grpc_tools_binary_host_mirror": "https://github.com/maschwenk/grpc-node/raw/2dd28e1ab8211533007dd2df5ae632de60006983/artifacts/",
    },
)

I figure I'll fork the grpc-node repo myself with Max's changes, so that I can build these artifacts myself when I need them updated for new versions. @maschwenk : could you clarify how mirrors work in the Yarn/NPM world? i.e. is the mirror only used if the central repository doesn't have the requested artifact? I ask because I want my code to support darwin-arm64 in addition to other architectures (for which there are grpc-utils artifacts already).

[maschwenk]

I figure I'll fork the grpc-node repo myself with Max's changes, so that I can build these artifacts myself when I need them updated for new versions.

😆 👍🏼 definitely don't recommend using internet random's binaries.

@maschwenk : could you clarify how mirrors work in the Yarn/NPM world? i.e. is the mirror only used if the central repository doesn't have the requested artifact?

So I think what's happening is actually:

1. https://docs.npmjs.com/cli/v8/using-npm/config#environment-variables only npm supports this conversion of flags prepended with npm_config to configure the process.env
2. node-pre-gyp is using the ENV or something set in the package.json to find the binary. I believe the default behavior is just using what's defined in the binary section of a package.json, so we're intentionally overriding it with the ENV
3. Because not everyone uses npm, you can just use a standard ENV injection to get # 2 to work

[dimi-nk]

I was able to build it on an Apple Silicon Mac like so: ...

Thanks, @maschwenk, that helped a lot. I'm missing the bit on how to upload the binaries to github.

To create the artifact, I tried both:

• npm pack inside grpc-node/packages/grpc-tools
• tar -zcvf darwin-arm64.tar.gz bin inside grpc-node/packages/grpc-tools/build .

I then upload each artifact (one at the time) into an empty repo under path artifacts/grpc-tools/v1.11.2/darwin-arm64.tar.gz but I'm getting TAR_BAD_ARCHIVE: Unrecognized archive format when trying to install using the env option. I'm wondering how you tarballed the files in the first place.

[stephennancekivell]

I'm using docker and npx as a workaround. It works pretty well.

docker run --rm --platform linux/amd64 \
        -v `pwd`:'/app' \
        -w '/app' node:16 npx \
        -y --package=grpc-tools \
        -- grpc_tools_node_protoc \
        --js_out=import_style=commonjs,binary:./nodejs/ --grpc_out=grpc_js:./nodejs/ ./*.proto

[spoo24]

I found a workaround by forcing terminal to x64 architecture and running the service

Commands

1. env /usr/bin/arch -x86_64 /bin/zsh --login
2. npm install
3. and start your service

Note that architecture preference is only for given terminal window.

[ravicious]

If anyone wants to know how to deal with this inside an arm64 Docker image running Ubuntu, please take a look at the final part of build.assets/Dockerfile-teleterm in my commit above that references this issue.

I added an arm64 CMake toolchain which is something that was missing. One thing to note is that while grpc-tools needs two binaries, protoc and grpc_node_plugin, our image already had protoc. If you want to compile both binaries, just remove the --target grpc_node_plugin flag from the final cmake call.

[sgammon]

this issue would never have surfaced if a pre-built ARM binary had simply been published

[sgammon]

most of the gRPC ecosystem builds on raw systems via Bazel and lower level tooling. why this doesn't, as an exception, makes no sense. surely there is somewhere within Google that isn't relying on Github Actions to use this code?

could google be so kind as to unbreak their node package ecosystem for most mac users?

for instance, is it possible to move this to CI infra where cross compiling is possible? or can node-gyp be supported to dynamically link against protoc on ARM, given that M1 has been out now for some time (November 2020), M2 is on the way, and x86 is not even for sale anymore for most Mac lines?

the entire point of gRPC is safe transport of data across systems which may differ in architecture. i don't understand what kind of barrier could possibly exist toward solving that problem; that is the reason this code exists

[sgammon]

also @srabraham thank you for posting bazel instructions here, that's very helpful

[pkpfr]

This isn't even an M1 issue - just needs to provide a build for aarch64 / arm64. It does not need to be compiled on an M1, and arm64 system will provide a working binary. Not only are M1 users being forced to use sub-optimal x86 builds, but the entire Raspberry Pie community have been neglected.