search

embarklabs / embark

Framework for serverless Decentralized Applications using Ethereum, IPFS and other platforms
https://framework.embarklabs.io/
MIT License
3.79k stars 495 forks source link

Remove dependency on node-gyp. #1995

Open MicahZoltu opened 5 years ago

MicahZoltu commented 5 years ago

Feature Request

Do not have any direct or transitive dependence on node-gyp.

Summary

Currenty, node-gyp is required to use Embark. This creates a notable barrier to entry because node-gyp depends on a ton of OS/architecture specific build tools including make, gcc, python, VC++, etc. (depending on OS). There is really no reason Embark needs node-gyp as all of Ethereum's crypto can be done with pure JS library alternatives.

Running into the transitive dependency on node-gyp has caused me to look elsewhere for frameworks to use, as in my experience any project with a dependency on node-gyp ends up causing me no end of headache. I'm filing this issue in an effort to encourage the Embark developers to drop the node-gyp dependency so that future users don't get turned away from the project like I have.

michaelsbradleyjr commented 5 years ago

We feel your pain. However, node-gyp is strictly required by several of our transitive dependencies, i.e. dependencies of our dependencies. This is a problem facing many libraries and frameworks in the Ethereum ecosystem since some of the cryptographic primitives required for Ethereum related things aren't provided by the runtime and pure-JS implementations tended to be slow. The situation is improving across the ecosystem, but it will be some time before we can fully eliminate all dependencies and transitive dependencies that rely on node-gyp for compiling crypotgraphy related functions implemented in C/C++.

MicahZoltu commented 5 years ago

Embark isn't running a miner, so I am dubious about the need for fast crypto here. Which libraries are you unable to get rid of due to transitive dependencies? In my experience, there are solutions available for solving most/all problems without having to pull in something with a transitive dependency on node-gyp. For example, ethereumjs-util is notoriously bad at pulling in unnecessarily complicated transitive dependencies, several of which require node-gyp, but a project can simply not use ethereumjs-util and instead use something else.

Note: I do acknowledge that migrating away from some of these bad libraries is not a trivial task, but it is one that it feels like is worth having on the roadmap and working toward over time. FWIW, I have tried lobbying to get some of the common ecosystem libraries to fix their packages and stop depending on node-gyp, but I haven't had much luck. My solution has been to just drop those dependencies in my projects in favor of something else and/or create alternatives where alternatives didn't exist.

michaelsbradleyjr commented 5 years ago

the need for fast crypto [in JavaScript contexts]

I'm generally agreed, as are some of the folks that work on EthereumJS libs and the maintainer of web3.js; and probably other lib/framework maintainers as well, but those are the ones I've communicated with directly regarding this very problem.

What we have in mind: libs should first check if the runtime, e.g. Node.js, provides a built-in KDF or hash, etc., and use it if it's available. If it's not available, then fallback to a pure-JS implementation. In some cases, the lib can also check if a native-module implementation is resolvable, e.g. with Node's require.resolve, and use that instead of the pure-JS implementation (in the case a built-in isn't available and the user elected to install such a module). I personally implemented such an approach for the scrypt KDF and it's used by web3@1.2.2.

Embark isn't running a miner

scrypt, keccak-256, and secp256k1 are examples of non-mining crypto facilities involved in Ethereum keystores, keys, and signing-verifying transactions. Regarding what I wrote above about "[w]hat we have in mind", the reality today is that many Ethereum-related libraries don't yet take that approach and instead rely on implementations that involve node-gyp.

The Embark team, as well as other lib/framework devs in this space, are hoping for and working toward an improved situation for the broader Ethereum-JavaScript ecosystem, but it will take a bit of time. The problem is amplified when one's dependency tree involves dependencies-of-dependencies-of-dependencies that are using older versions of libraries (that depend on node-gyp) and the "fork to solve" approach of solving this kind of problem gets much messier.