search

grpc / grpc-node

gRPC for Node.js
https://grpc.io
Apache License 2.0
4.49k stars 651 forks source link

missing pre-compiled arm binary #1405

Closed shotor closed 1 year ago

shotor commented 4 years ago

Problem description

Can't install grpc-tools on arm architecture (raspberry pi) because of missing precompiled binaries.

node-pre-gyp ERR! stack Error: 404 status code downloading tarball https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz

Is it possible to build these at all?

Reproduction steps

npm init npm install grpc-tools

Environment

Additional context

error /home/repo/grpc/node_modules/grpc-tools: Command failed.
Exit code: 1
Command: node-pre-gyp install
Arguments:
Directory: /home/repo/grpc/node_modules/grpc-tools
Output:
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using node-pre-gyp@0.12.0
node-pre-gyp info using node@10.20.1 | linux | arm64
node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp info check checked for "/home/repo/grpc/node_modules/grpc-tools/bin/grpc_tools.node" (not found)
node-pre-gyp http GET https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
node-pre-gyp http 404 https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
node-pre-gyp ERR! install error
node-pre-gyp ERR! stack Error: 404 status code downloading tarball https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
node-pre-gyp ERR! stack     at Request.<anonymous> (/home/repo/grpc/node_modules/node-pre-gyp/lib/install.js:149:27)
node-pre-gyp ERR! stack     at Request.emit (events.js:203:15)
node-pre-gyp ERR! stack     at Request.onRequestResponse (/home/repo/grpc/node_modules/request/request.js:1059:10)
node-pre-gyp ERR! stack     at ClientRequest.emit (events.js:198:13)
node-pre-gyp ERR! stack     at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:565:21)
node-pre-gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:111:17)
node-pre-gyp ERR! stack     at TLSSocket.socketOnData (_http_client.js:451:20)
node-pre-gyp ERR! stack     at TLSSocket.emit (events.js:198:13)
node-pre-gyp ERR! stack     at addChunk (_stream_readable.js:288:12)
node-pre-gyp ERR! stack     at readableAddChunk (_stream_readable.js:269:11)
node-pre-gyp ERR! System Linux 5.4.0-1008-raspi
node-pre-gyp ERR! command "/usr/local/bin/node" "/home/repo/grpc/node_modules/grpc-tools/node_modules/.bin/node-pre-gyp" "install"
node-pre-gyp ERR! cwd /home/repo/grpc/node_modules/grpc-tools
node-pre-gyp ERR! node -v v10.20.1
node-pre-gyp ERR! node-pre-gyp -v v0.12.0
node-pre-gyp ERR! not ok
murgatroid99 commented 2 years ago

We don't provide Linux arm64 binaries for the same reason that we don't provide Mac M1 binaries: we don't have the environment available to do the build. Protobuf is a pain to cross-compile because of how it bootstraps itself, so we are only providing binaries that we can build on the corresponding environment. M1 build machines are at least on the roadmap for GitHub Actions, so we expect to support them eventually, but Linux arm64 is not on that roadmap, so we have no plans to provide Linux arm64 binaries.

If you want to use a Raspberry Pie, I suggest performing the build on another machine and then transferring the output files to the Pie. The grpc-tools package is just a build tool; it is not needed at runtime.

Mills-2 commented 2 years ago

@murgatroid99 Is this the item that is blocking the arm64 pipeline? If so it just shipped out and I just wanted to point it out and potentially give an update on this for the community.

Disregard that looks like it was the wrong item after doing some more reading. My bad.

glenjamin commented 2 years ago

We were able to do a local build, upload that to an S3 bucket, and then configure npm to look there for binaries.

Here's the rough steps to follow

We also did the same thing for grpc-node's native package, only that was easier to build as we could use the node-pre-gyp automatic source fallback and then node-pre-gyp package to create the tarball.

janpieterz commented 2 years ago

@Mills-2 Could you expand on why it wouldn't work? I did a local compile on my M1, and it works if I use @glenjamin's solution, but before diving into this and providing a PR I'd love to know what the blocker would have been.

glenjamin commented 2 years ago

My understanding is that the project doesn't want to provide pre-compiled binaries that were not created by a publicly visible CI provider.

So while it's possible to create binaries in an ad-hoc manner on individual M1 machines, there isn't yet a good way to hook this into the automated build tooling for the project.

janpieterz commented 2 years ago

@glenjamin yea that makes sense, but Github's M1 agents that @Mills-2 linked to would hypothetically work?

Thanks for the suggestion btw, @glenjamin, works great for us so far.

Mills-2 commented 2 years ago

@janpieterz The issue I linked is for self-hosted runners. This issue is blocked by github hosted runners. It is still in future implementation on the roadmap for github.

briangriffeyudemy commented 2 years ago

I'm happy to donate an arm mac to the grpc team for the self hosted runner.

murgatroid99 commented 2 years ago

Version 1.11.3 now has Mac M1/arm64 binaries.

loudmouth commented 2 years ago

A great notification to wake up to. Thanks to all the people who made that happen.

Any chance the latest version can be published to NPM as well?

Edit: was just looking at the wrong package @grpc/grpc-js. Of course this was about grpc-tools

murgatroid99 commented 2 years ago

The latest version is 1.11.3 and it is on NPM.

maschwenk commented 2 years ago

@murgatroid99 I updated to that new version and it worked a treat. Thank you!

hexnaught commented 2 years ago

Version 1.11.3 now has Mac M1/arm64 binaries.

Updated too and works great now without any work arounds. Thank you! 🙏

pballester commented 2 years ago

Still missing linux arm64 binary 😢

https://node-precompiled-binaries.grpc.io/grpc-tools/v1.11.3/linux-arm64.tar.gz

ghost commented 2 years ago

Version 1.11.3 now has Mac M1/arm64 binaries.

Still not available for linux (exp: if you have a docker image):

response status 404 Not Found on https://node-precompiled-binaries.grpc.io/grpc-tools/v1.11.3/linux-arm64.tar.gz

so I've added the following to my dockerfile

RUN npm_config_target_arch=x64 npm i grpc-tools
ravicious commented 2 years ago

@source-transformer Excuse me, but how does this work? Assuming you have an arm64 Linux image, how is it able to run x64 binaries? Wouldn't that result in an error anyway?

Or do you just install the package but not actually use its components which require x64 binaries?

ghost commented 2 years ago

Or do you just install the package but not actually use its components which require x64 binaries?

Correct - I should have qualified my answer with - you can't actually run grpc-tools on your container. I had tried the following:

npm_config_target_arch=x86 npm i grpc-tools

In the hopes of getting a binary that potentially could be run. But, honestly that was just a shot in the dark as to the correct label/alias for the 32 bit build/architecture. I wasn't sure how to get a listing of supported architectures.

In my case - it isn't necessary to actually run grpc-tools on my docker container - since those tools are "devDependencies" and their output is actually copied to the docker image.

huangjun0210 commented 1 year ago

Will it support linux arm64 in the future? I'm looking forward to it.

RedstoneWizard08 commented 1 year ago

A while back we made an alternative mirror for this... https://github.com/ZCube/grpc-precompiled-binaries

huangjun0210 commented 1 year ago

A while back we made an alternative mirror for this... https://github.com/ZCube/grpc-precompiled-binaries

How to bulid linux arm64? when I follow with this document: https://github.com/grpc/grpc-node/blob/master/packages/grpc-tools/build_binaries.sh。Error occurred：

...
[ 99%] Linking CXX executable protoc
[100%] Linking CXX executable grpc_node_plugin
/usr/bin/ld: cannot find -lstdc++
collect2: error: ld returned 1 exit status
gmake[2]: *** [grpc_node_plugin] Error 1
gmake[1]: *** [CMakeFiles/grpc_node_plugin.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[100%] Built target protoc
gmake: *** [all] Error 2

ldconfig -p | less 

        libstdc++.so.6 (libc6,AArch64) => /lib64/libstdc++.so.6
        libstdc++.so (libc6,AArch64) => /usr/local/lib/libstdc++.so
        libssl3.so (libc6,AArch64) => /lib64/libssl3.so
RedstoneWizard08 commented 1 year ago

I'll have to look again, it's been a while. I know it was pretty simple, though.

geminiyellow commented 1 year ago

2 years later this issue still open 

node-pre-gyp info it worked if it ends with ok | 88s
-- | --
86 | node-pre-gyp info using node-pre-gyp@1.0.10 | 88s
87 | node-pre-gyp info using node@16.17.1 \| linux \| x64 | 88s
88 | node-pre-gyp info check checked for "/drone/src/node_modules/grpc-tools/bin/grpc_tools.node" (not found) | 88s
89 | node-pre-gyp http GET https://node-precompiled-binaries.grpc.io/grpc-tools/v1.12.3/linux-x64.tar.gz | 88s
90 | node-pre-gyp ERR! install request to https://node-precompiled-binaries.grpc.io/grpc-tools/v1.12.3/linux-x64.tar.gz failed, reason: read ECONNRESET | 88s
91 | node-pre-gyp ERR! install error
fatihky commented 1 year ago

@murgatroid99 and other contributors, are you planning to add support for the arm architecture? if you don't have enough time for that, I'd like to try to add arm support to your library.

murgatroid99 commented 1 year ago

I have no immediate plans to add this myself. If someone contributes a build change that produces ARM binaries for grpc-tools, I will accept it and publish them.

RedstoneWizard08 commented 1 year ago

As soon as I remember how I did it, I'll integrate it into the build system and make a PR. (I hope that this message showing up multiple times is just a glitch in the GitHub app on my phone.)

RedstoneWizard08 commented 1 year ago

@murgatroid99 I just finished a patch. https://github.com/RedstoneWizard08/grpc-node Successful build: https://github.com/RedstoneWizard08/grpc-node/actions/runs/3981949215/jobs/6826020346

PR: #2333

RedstoneWizard08 commented 1 year ago

Merged!

murgatroid99 commented 1 year ago

The Linux ARM64 binary is now available in version 1.12.4.

yurks commented 1 year ago

Version 1.11.3 now has Mac M1/arm64 binaries.

actually it hasn't: https://github.com/grpc/grpc-node/issues/2378

MehmetSecgin commented 1 year ago

Is it planned to support M2 architecture as well?

rburdicksuki commented 4 months ago

The latest version is 1.11.3 and it is on NPM.

Which repo are we talking about here? It sounds like folks have gotten grpc-node to work on Mac M series, is there a concise set of instructions I can follow?