search

holepunchto / hyperswarm

A distributed networking stack for connecting peers.
https://docs.holepunch.to
MIT License
1.06k stars 85 forks source link

electron-rebuild: failed to rebuild node_modules/@hyperswarm/dht/node_modules/sodium-native #70

Open raphael10-collab opened 3 years ago

raphael10-collab commented 3 years ago

In my electron-react-typescript project I added hyperswarm:

yarn add hyperswarm
yarn add v1.22.5
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents@2.1.3: The platform "linux" is incompatible with this module.
info "fsevents@2.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
info fsevents@1.2.13: The platform "linux" is incompatible with this module.
info "fsevents@1.2.13" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "ipfs-core > native-abort-controller@0.0.3" has unmet peer dependency "abort-controller@*".
warning "react-bootstrap > @restart/hooks@0.3.25" has incorrect peer dependency "react@^16.8.0".
warning " > react-pdf@5.0.0" has incorrect peer dependency "react@^16.3.0".
warning " > react-pdf@5.0.0" has incorrect peer dependency "react-dom@^16.3.0".
warning "webpack-dev-server > webpack-dev-middleware@3.7.2" has incorrect peer dependency "webpack@^4.0.0".
warning "wrtc@0.4.6" is missing a bundled dependency "node-pre-gyp". This should be reported to the package maintainer.
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 14 new dependencies.
info Direct dependencies
└─ hyperswarm@2.15.2
info All dependencies
├─ @hyperswarm/dht@4.0.1
├─ @hyperswarm/discovery@2.0.1
├─ @hyperswarm/hypersign@2.1.1
├─ @hyperswarm/network@2.1.0
├─ blake2b-universal@1.0.1
├─ dht-rpc@4.9.6
├─ guard-timeout@2.0.0
├─ hyperswarm@2.15.2
├─ ipv4-peers@2.0.0
├─ record-cache@1.1.0
├─ shuffled-priority-queue@2.1.0
├─ speedometer@1.1.0
├─ stream-collector@1.0.1
└─ time-ordered-set@1.0.2
Done in 9.31s.

When electron-rebuild-ing : 

(base) marco@pc01:~/webMatters/electronMatters/GGC$ $(npm bin)/electron-rebuild
⠧ Building module: sodium-native, Completed: 0gyp info find Python using Python version 3.7.4 found at "/home/marco 
/anaconda3/bin/python3"
⠇ Building module: sodium-native, Completed: 0gyp http GET https://www.electronjs.org/headers/v11.0.3/node-v11.0.3-
headers.tar.gz
⠇ Building module: sodium-native, Completed: 0gyp http 200 https://www.electronjs.org/headers/v11.0.3/node-v11.0.3-
headers.tar.gz
⠙ Building module: sodium-native, Completed: 0gyp http GET https://www.electronjs.org/headers/v11.0.3/SHASUMS256.txt
⠙ Building module: sodium-native, Completed: 0gyp http 200 https://www.electronjs.org/headers/v11.0.3/SHASUMS256.txt
gyp info spawn /home/marco/anaconda3/bin/python3
gyp info spawn args [
gyp info spawn args   '/home/marco/webMatters/electronMatters/GGC/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/home/marco/webMatters/electronMatters/GGC/node_modules/@hyperswarm/dht/node_modules
/sodium-native/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/marco/webMatters/electronMatters/GGC/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/marco/webMatters/electronMatters/GGC/node_modules/@hyperswarm/dht/node_modules
/sodium-native/11.0.3/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/marco/webMatters/electronMatters/GGC/node_modules/@hyperswarm
/dht/node_modules/sodium-native/11.0.3',
gyp info spawn args   '-Dnode_gyp_dir=/home/marco/webMatters/electronMatters/GGC/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/marco/webMatters/electronMatters/GGC/node_modules/@hyperswarm 
/dht/node_modules/sodium-native/11.0.3/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/home/marco/webMatters/electronMatters/GGC/node_modules/@hyperswarm
/dht/node_modules/sodium-native',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
⠸ Building module: sodium-native, Completed: 0gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/home/marco/webMatters/electronMatters/GGC/node_modules/@hyperswarm/dht/node_modules
/sodium-native/build'
  CC(target) Release/obj.target/sodium/binding.o
In file included from ../binding.c:5:
../libsodium/src/libsodium/include/sodium.h:5:10: fatal error: sodium/version.h: No such file or directory
    5 | #include "sodium/version.h"
     |            ^~~~~~~~~~~~~~~~~~
compilation terminated.
sodium.target.mk:123: recipe for target 'Release/obj.target/sodium/binding.o' failed
make: *** [Release/obj.target/sodium/binding.o] Error 1
make: Leaving directory '/home/marco/webMatters/electronMatters/GGC/node_modules/@hyperswarm/dht/node_modules
/sodium-native/build'
✖ Rebuild Failed

An unhandled error occurred inside electron-rebuild
node-gyp failed to rebuild '/home/marco/webMatters/electronMatters/GGC/node_modules/@hyperswarm/dht/node_modules   
/sodium-native'.
Error: `make` failed with exit code: 2

Error: node-gyp failed to rebuild '/home/marco/webMatters/electronMatters/GGC/node_modules/@hyperswarm  
/dht/node_modules/sodium-native'.
Error: `make` failed with exit code: 2

    at ModuleRebuilder.rebuildNodeGypModule (/home/marco/webMatters/electronMatters/GGC/node_modules/electron-rebuild
/lib/src/module-rebuilder.js:193:19)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Rebuilder.rebuildModuleAt (/home/marco/webMatters/electronMatters/GGC/node_modules/electron-rebuild/lib/src
/rebuild.js:190:9)
    at async Rebuilder.rebuild (/home/marco/webMatters/electronMatters/GGC/node_modules/electron-rebuild/lib/src
/rebuild.js:152:17)
    at async /home/marco/webMatters/electronMatters/GGC/node_modules/electron-rebuild/lib/src/cli.js:146:9

Other info:

node version: v.14.5.0
"devDependencies": {
  "@types/react": "^17.0.0",
  "@types/react-dom": "^17.0.0",
  "electron: "11.0.2"
 },
"dependencies": {
  "react": "^17.0.1",
   "react-dom": "^17.0.1"
 }

How to solve the problem?

mafintosh commented 3 years ago

@raphael10-collab is the sodium prebuild not working for you? normally you'd never rebuild sodium

raphael10-collab commented 3 years ago

Hi @mafintosh !

The problem comes from here:

(base) marco@pc01:~/webMatters/electronMatters/GGC$ yarn start
yarn run v1.22.5
$ npm run build && electron ./dist/main.js

> GGC@1.0.0 build /home/marco/webMatters/electronMatters/GGC
> tsc

ℹ️  ggc/desktop/ConfigurationPersistence [2020-12-02 19:11:32] Reading config file from "/home/marco/.config/GGC/config
/init.json" ...
ℹ️  ggc/desktop/ConfigurationPersistence [2020-12-02 19:11:32] Inizializing ConfigurationPersistence
ℹ️  ggc/desktop/ConfigurationPersistence [2020-12-02 19:11:32] Restore "locale"
ℹ️  ggc/desktop/ConfigurationPersistence [2020-12-02 19:11:32] Restore "env"
ℹ️  ggc/desktop/ConfigurationPersistence [2020-12-02 19:11:32] Restore "enableSpellChecking" 
ℹ️  ggc/desktop/ConfigurationPersistence [2020-12-02 19:11:32] Restore "showMenu"
ℹ️  ggc/desktop/ConfigurationPersistence [2020-12-02 19:11:32] Restore "shouldAutoLaunch"
App threw an error during load
Error: The module '/home/marco/webMatters/electronMatters/GGC/node_modules/sodium-native/build/Release/sodium.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 83. This version of Node.js requires
NODE_MODULE_VERSION 85. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
    at process.func [as dlopen] (electron/js2c/asar_bundle.js:5:1812)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1203:18)
    at Object.func [as .node] (electron/js2c/asar_bundle.js:5:1812)
    at Module.load (internal/modules/cjs/loader.js:992:32)
    at Module._load (internal/modules/cjs/loader.js:885:14)
    at Function.f._load (electron/js2c/asar_bundle.js:5:12694)
    at Module.require (internal/modules/cjs/loader.js:1032:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at load (/home/marco/webMatters/electronMatters/GGC/node_modules/node-gyp-build/index.js:21:10)
    at Object.<anonymous> (/home/marco/webMatters/electronMatters/GGC/node_modules/sodium-native/index.js:1:39)
A JavaScript error occurred in the main process
Uncaught Exception:
Error: The module '/home/marco/webMatters/electronMatters/GGC/node_modules/sodium-native/build/Release/sodium.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 83. This version of Node.js requires
NODE_MODULE_VERSION 85. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
at process.func [as dlopen] (electron/js2c/asar_bundle.js:5:1812)
at Object.Module._extensions..node (internal/modules/cjs/loader.js:1203:18)
at Object.func [as .node] (electron/js2c/asar_bundle.js:5:1812)
at Module.load (internal/modules/cjs/loader.js:992:32)
at Module._load (internal/modules/cjs/loader.js:885:14)
at Function.f._load (electron/js2c/asar_bundle.js:5:12694)
at Module.require (internal/modules/cjs/loader.js:1032:19)
at require (internal/modules/cjs/helpers.js:72:18)
at load (/home/marco/webMatters/electronMatters/GGC/node_modules/node-gyp-build/index.js:21:10)
at Object.<anonymous> (/home/marco/webMatters/electronMatters/GGC/node_modules/sodium-native/index.js:1:39)

This is why I tried to use electron-rebuild.

mafintosh commented 3 years ago

That looks like a pretty old version of sodium, can you check somehow?

raphael10-collab commented 3 years ago

Sorry for the question... how to check the version of the sodium lib? I didn't find any reference in /node_modules/node-gyp/package.json and /node_modules/hyperswarm/package

mafintosh commented 3 years ago

You can see the path in the problem you posted here https://github.com/hyperswarm/hyperswarm/issues/70#issuecomment-737438666, ie /home/marco/webMatters/electronMatters/GGC/node_modules/sodium-native/package.json

mafintosh commented 3 years ago

But I see now also that, that dep was rebuilt as well. In general don't rebuild sodium, utp-native or any of the prebuilds we ship. It'll create a lot more problems like this with interop than if you don't :)

raphael10-collab commented 3 years ago

/home/marco/webMatters/electronMatters/GGC/node_modules/sodium-native/package.json : 

"version": "2.4.9"

Do you think that upgrading sodium-native to the latest version would solve the problem, or should I only re-install hyperswarm?

mafintosh commented 3 years ago

Yes, very likely

On 2 Dec 2020, at 20.37, raphael10-collab notifications@github.com wrote:

 /home/marco/webMatters/electronMatters/GGC/node_modules/sodium-native/package.json :

"version": "2.4.9" Do you think that upgrading sodium-native to the latest version would solve the problem?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

raphael10-collab commented 3 years ago

Now /home/marco/webMatters/electronMatters/GGC/node_modules/sodium-native/package.json : 

"version": "3.2.0"

Still get:

App threw an error during load
Error: The module '/home/marco/webMatters/electronMatters/GGC/node_modules/noise-protocol/node_modules/sodium-native
/build/Release/sodium.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 83. This version of Node.js requires
NODE_MODULE_VERSION 85. Please try re-compiling or re-installing

I removed node_modules folder and yarn.lock file. Then I re-installed all the packages. But still the same problem.

derhuerst commented 3 years ago

Could it be that you updated sodium-native to the latest version but noise-protocol has its own sodium-native dependency? npm usually does this whenever you want a version of package that is incompatible with the version that a dependency wants; In this case, npm nests the dependency's package. You can use npm ls to see the version of the nested dependency, maybe this will help.

mafintosh commented 3 years ago

Latest noise-protocol uses latest sodium also. All of the hyper* deps are on sodium 3.

mafintosh commented 3 years ago

But since you have a build folder inside sodium above, something is still rebuilding. Did you clean out all your deps and reinstalled? You sure nothing is compiling or that you aren't recompiling anything?

When you recompile you loose the cross node/electron compat that the prebuilds give you.

raphael10-collab commented 3 years ago

This is the complete list of dependencies obtained through yarn list command: yarnList.txt I'm trying to distill from your kind suggestions the steps I need to follow.

From the complete list of dependencies I see that sodium is version 3.something in these packages: @hyperswarm/dht@4.0.1 , @hyperswarm/hypersign@2.1.1, blake2b-universal@1.0.1, dht-rpc@4.9.6 .

But it is version 2.something as dependency in other packages: hmac-blake2b@0.2.0 , hypercore-crypto@1.0.0 hypercore@8.14.0 hypermerge@2.0.0-beta.25 noise-peer@1.1.0 noise-protocol@1.0.0 secretstream-stream@1.2.1 simple-hypercore-protocol@1.5.0 sodium-universal@2.0.0

So...should I remove the sodium dependency of each package if it is version 2.something, and re-install it @latest or should I do something else?

mafintosh commented 3 years ago

What's pulling in those deps with version 2? Nothing in our stack should be doing that.

raphael10-collab commented 3 years ago

Sorry Mathias. May be to my lack of knowledge, I do not understand. Should I upgrade every sodium package to version 3?

mafintosh commented 3 years ago

@raphael10-collab no problem, let me try to be a bit more clear.

Some parent dep in your dependency tree in your project is pulling in a legacy dep that uses sodium 2. For me to easier help you i need to know what parent dep that is. Could you post the output of npm ls from your project? Then I can take a look :)

raphael10-collab commented 3 years ago

@mafintosh this is the complete dependencies tree: yarnList.txt

mafintosh commented 3 years ago

I think that list is deduped, but I think hypermerge in there needs a hypercore major bump. We should make an issue for that on that repo. You are using that one yea?

raphael10-collab commented 3 years ago

I now redone the same list with this command: yarn list > yarnList.txt : yarnList.txt

Yes, I'm using hypercore. I can make an issue in github for hypercore's library asking for upgrading their sodium dependencies. And it would be helpful if you can give them your voice as well

mafintosh commented 3 years ago

It's not hypercore, Hypercore is using latest sodium. It's the dep that's using an old version of Hypercore that needs to be bumped to use latest Hypercore, among those is Hypermerge (which we don't maintain)

raphael10-collab commented 3 years ago

@mafintosh I just made an issue in Hypermerge's GitHub page about the version of Hypercore used: https://github.com/automerge/hypermerge/issues/88