search

laurent22 / joplin

Joplin - the privacy-focused note taking app with sync capabilities for Windows, macOS, Linux, Android and iOS.
https://joplinapp.org
Other
45.85k stars 4.97k forks source link

Dependency issues when building on Ubuntu x64 #4385

Closed elsiehupp closed 3 years ago

elsiehupp commented 3 years ago

I have reams and reams of logs from npm install failing, but ultimately doing the following seemed to fix the problem:

  1. sudo apt install python

    This was because node-sqlite3 didn't have a binary for Ubuntu x64, and building it from scratch required python, i.e. python2-as-python.

  2. sudo npm install --global jetifier sudo npm install --global gulp sudo npm install --global rollup

I first ran sudo npm install --global lerna and then ran lerna bootstrap --no-ci to find each of these broken dependencies. lerna could not find any of these three packages unless they were installed globally, so perhaps some aspect of the build script is not calling them correctly for when they are installed only within the project directory.

elsiehupp commented 3 years ago

It also gets angry if you run npm install in packages/app-desktop instead of in the root, and the resulting error message is inscrutable.

laurent22 commented 3 years ago

We don't support most of what you wrote about, such as direct Lerna commands or running npm i from within a package. Best is to follow the instructions BUILD.md as anything else is likely to fail.

elsiehupp commented 3 years ago

Best is to follow the instructions BUILD.md as anything else is likely to fail.

The reason I posted this issue is that I followed the instructions in BUILD.md, and it didn't work.

As for running direct lerna commands, I literally just copied and pasted the command from the build script so that the debug output would appear directly in my terminal rather than in a log file somewhere. Direct lerna commands (and installing lerna globally) were not strictly necessary for making the build work, just for debugging in order to figure out how to do so.

Again, I followed the instructions in BUILD.md, and npm install failed because of issues with the dependencies I listed above. The lerna build process as called by npm install could not find jetify (i.e. jetifier, gulp, or rollup unless I installed them globally. I can intentionally break my setup again so that I can dump the log files on you if you really insist.

I see that you removed the "bug" label. While this issue isn't a bug with the built software, it is a bug with the build script. It is, indeed, actually a bug, which you will see if you more closely reread what I wrote.

elsiehupp commented 3 years ago

Oh, look, I'm getting build errors on my Mac, as well:

% npm install

[extraneous output removed]

lerna info lifecycle @joplin/app-clipper@1.0.8~postinstall: Failed to exec postinstall script

> @joplin/app-mobile@0.8.9 postinstall /Users/elsiehupp/Repositories/joplin/packages/app-mobile
> jetify && npm run build

npm ERR! code 1
npm ERR! path /Users/elsiehupp/Repositories/joplin
npm ERR! command failed
npm ERR! command sh -c lerna bootstrap --no-ci

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/elsiehupp/.npm/_logs/2021-01-18T01_10_23_822Z-debug.log
npm ERR! code 1
npm ERR! path /Users/elsiehupp/Repositories/joplin
npm ERR! command failed
npm ERR! command sh -c npm run bootstrap --no-ci && npm run build

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/elsiehupp/.npm/_logs/2021-01-18T01_10_23_903Z-debug.log

Huh!

If I go into the first log file, this is what I see:

22 verbose stack Error: command failed
22 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/index.js:64:27)
22 verbose stack     at ChildProcess.emit (node:events:376:20)
22 verbose stack     at maybeClose (node:internal/child_process:1063:16)
22 verbose stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:295:5)
23 verbose cwd /Users/elsiehupp/Repositories/joplin
24 verbose Darwin 20.2.0
25 verbose argv "/usr/local/Cellar/node/15.5.0/bin/node" "/usr/local/bin/npm" "run" "bootstrap" "--no-ci"
26 verbose node v15.5.0
27 verbose npm  v7.3.0
28 error code 1
29 error path /Users/elsiehupp/Repositories/joplin
30 error command failed
31 error command sh -c lerna bootstrap --no-ci

If I go into the second log file, this is what I see:


70 verbose stack Error: command failed
70 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/index.js:64:27)
70 verbose stack     at ChildProcess.emit (node:events:376:20)
70 verbose stack     at maybeClose (node:internal/child_process:1063:16)
70 verbose stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:295:5)
71 verbose cwd /Users/elsiehupp/Repositories/joplin
72 verbose Darwin 20.2.0
73 verbose argv "/usr/local/Cellar/node/15.5.0/bin/node" "/usr/local/bin/npm" "install"
74 verbose node v15.5.0
75 verbose npm  v7.3.0
76 error code 1
77 error path /Users/elsiehupp/Repositories/joplin
78 error command failed
79 error command sh -c npm run bootstrap --no-ci && npm run build

Googling "@npmcli/promise-spawn/index.js:64:27", I get this Stack Overflow result, which suggests the problem is a typo somewhere in package.json.

elsiehupp commented 3 years ago

Incidentally, while npm install doesn't work on my Mac (that is, in addition to outputting errors, it doesn't actually create a fresh Joplin build), npm run build does work (and does create a fresh Joplin build). I have no idea why this is.

laurent22 commented 3 years ago

What's the complete output of npm i?

As for running direct lerna commands, I literally just copied and pasted the command from the build script

You've installed Lerna globally and ran that, so you're most likely using a version that's different from the one that comes with the monorepo. You've also installed it with sudo, which you shouldn't as it will cause various permission issues. If you want to use Lerna commands, you need to run them as mentioned in BUILD.md: npx lerna ...

laurent22 commented 3 years ago

I'm going to close the issue since it's not a bug. It's just that sometimes it's tricky to setup Node, npm, etc. correctly.

elsiehupp commented 3 years ago

Wow, you didn't even wait for me to respond!

Anyway, here are the currently globally installed packages on my Mac:

% npm list -g --depth=0
/usr/local/lib
└── npm@7.3.0

And here is the relevant portion of npm -i:

lerna info lifecycle @joplin/app-clipper@1.0.8~postinstall: Failed to exec postinstall script

> @joplin/app-desktop@1.7.0 postinstall /Users/elsiehupp/Repositories/joplin/packages/app-desktop
> npm run build

npm ERR! code 1
npm ERR! path /Users/elsiehupp/Repositories/joplin
npm ERR! command failed
npm ERR! command sh -c lerna bootstrap --no-ci

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/elsiehupp/.npm/_logs/2021-01-18T20_12_34_559Z-debug.log
npm ERR! code 1
npm ERR! path /Users/elsiehupp/Repositories/joplin
npm ERR! command failed
npm ERR! command sh -c npm run bootstrap --no-ci && npm run build

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/elsiehupp/.npm/_logs/2021-01-18T20_12_34_628Z-debug.log

(Note: it's pretty much exactly the same as the above.)

On my Linux computer:

$ npm list -g --depth=0
/usr/lib
├── gulp@4.0.2
├── jetifier@1.6.6
├── lerna@3.22.1
├── npm@6.14.10
└── rollup@2.36.2

npm ERR! peer dep missing: @octokit/core@>=3, required by @octokit/plugin-request-log@1.0.2

npm i currently runs without error.

Now I run the following: $ sudo npm remove gulp $ sudo npm remove jetifier $ sudo npm remove lerna $ sudo npm remove rollup $ sudo apt remove python $ sudo apt remove python-is-python2 $ cd .. $ sudo rm -r joplin $ git clone https://github.com/laurent22/joplin.git $ cd joplin $ git checkout 59fe4a21937111425c0296c1b0022a2fb146a1ee (This is the last commit from yesterday, before you fixed the error that you insisted was my fault.)

$ npm i

> root@ postinstall /home/elsiehupp/Repositories/joplin
> npm run bootstrap --no-ci && npm run build

> root@ bootstrap /home/elsiehupp/Repositories/joplin
> lerna bootstrap --no-ci

lerna notice cli v3.22.1
lerna info versioning independent
lerna info Bootstrapping 14 packages
lerna info Installing external dependencies
lerna ERR! npm install exited 1 in '@joplin/server'
lerna ERR! npm install stdout:

> sqlite3@4.1.0 install /home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

make: Entering directory '/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/build'
  ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlite-autoconf-3280000/sqlite3.c
make: Leaving directory '/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/build'
Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64 --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83' (1)

lerna ERR! npm install stderr:
node-pre-gyp WARN Using request for node-pre-gyp https download 
node-pre-gyp WARN Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v4.1.0/node-v83-linux-x64.tar.gz 
node-pre-gyp WARN Pre-built binaries not found for sqlite3@4.1.0 and node@14.15.4 (node-v83 ABI, glibc) (falling back to source compile with node-gyp) 
/bin/sh: 1: python: not found
make: *** [deps/action_before_build.target.mk:13: Release/obj/gen/sqlite-autoconf-3280000/sqlite3.c] Error 127
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
gyp ERR! System Linux 5.8.0-36-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64" "--napi_version=7" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v83"
gyp ERR! cwd /home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3
gyp ERR! node -v v14.15.4
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok 
node-pre-gyp ERR! build error 
node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64 --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/home/elsiehupp/Repositories/joplin/packages/server/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:1048:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)
node-pre-gyp ERR! System Linux 5.8.0-36-generic
node-pre-gyp ERR! command "/usr/bin/node" "/home/elsiehupp/Repositories/joplin/packages/server/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3
node-pre-gyp ERR! node -v v14.15.4
node-pre-gyp ERR! node-pre-gyp -v v0.11.0
node-pre-gyp ERR! not ok 
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.2.1 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.2.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.3 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! sqlite3@4.1.0 install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the sqlite3@4.1.0 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-18T21_07_35_306Z-debug.log

lerna ERR! npm install exited 1 in '@joplin/server'
lerna WARN complete Waiting for 7 child processes to exit. CTRL-C to exit immediately.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! root@ bootstrap: `lerna bootstrap --no-ci`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the root@ bootstrap script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-18T21_07_35_481Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! root@ postinstall: `npm run bootstrap --no-ci && npm run build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the root@ postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-18T21_07_35_535Z-debug.log

Oh, look! All the same errors came back! $ git checkout dev

$ npm i

> root@ postinstall /home/elsiehupp/Repositories/joplin
> npm run bootstrap --no-ci && npm run build

> root@ bootstrap /home/elsiehupp/Repositories/joplin
> lerna bootstrap --no-ci

lerna notice cli v3.22.1
lerna info versioning independent
lerna info Bootstrapping 15 packages
lerna info Installing external dependencies
lerna ERR! npm install exited 1 in '@joplin/server'
lerna ERR! npm install stdout:

> sqlite3@4.1.0 install /home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

make: Entering directory '/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/build'
  ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlite-autoconf-3280000/sqlite3.c
make: Leaving directory '/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/build'
Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64 --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83' (1)

lerna ERR! npm install stderr:
node-pre-gyp WARN Using request for node-pre-gyp https download 
node-pre-gyp WARN Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v4.1.0/node-v83-linux-x64.tar.gz 
node-pre-gyp WARN Pre-built binaries not found for sqlite3@4.1.0 and node@14.15.4 (node-v83 ABI, glibc) (falling back to source compile with node-gyp) 
/bin/sh: 1: python: not found
make: *** [deps/action_before_build.target.mk:13: Release/obj/gen/sqlite-autoconf-3280000/sqlite3.c] Error 127
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
gyp ERR! System Linux 5.8.0-36-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64" "--napi_version=7" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v83"
gyp ERR! cwd /home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3
gyp ERR! node -v v14.15.4
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok 
node-pre-gyp ERR! build error 
node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64 --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/home/elsiehupp/Repositories/joplin/packages/server/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:1048:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)
node-pre-gyp ERR! System Linux 5.8.0-36-generic
node-pre-gyp ERR! command "/usr/bin/node" "/home/elsiehupp/Repositories/joplin/packages/server/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3
node-pre-gyp ERR! node -v v14.15.4
node-pre-gyp ERR! node-pre-gyp -v v0.11.0
node-pre-gyp ERR! not ok 
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.3 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.2.1 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.2.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! sqlite3@4.1.0 install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the sqlite3@4.1.0 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-18T21_10_00_757Z-debug.log

lerna ERR! npm install exited 1 in '@joplin/server'
lerna WARN complete Waiting for 7 child processes to exit. CTRL-C to exit immediately.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! root@ bootstrap: `lerna bootstrap --no-ci`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the root@ bootstrap script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-18T21_10_00_823Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! root@ postinstall: `npm run bootstrap --no-ci && npm run build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the root@ postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-18T21_10_00_876Z-debug.log

Hrm, I guess it's not actually fixed!

Oh, and just to confirm that I'm actually on the current commit:

$ git show --oneline -s
9c718baf6 (HEAD -> dev, origin/dev, origin/HEAD) Merge branch 'dev' of github.com:laurent22/joplin into dev

Your most recent commit, earlier today:

https://github.com/laurent22/joplin/commit/9c718baf61a4a353592a9e53e7f0fcbc00b66d0a

EDIT: FYI I'm running Node 14.15.4 LTS, if that helps you reproduce the errors.

elsiehupp commented 3 years ago

I just completely uninstalled and then reinstalled nodejs, using the binary download rather than apt this time, and I'm getting exactly the same output:

$ npm i

> root@ postinstall /home/elsiehupp/Repositories/joplin
> npm run bootstrap --no-ci && npm run build

> root@ bootstrap /home/elsiehupp/Repositories/joplin
> lerna bootstrap --no-ci

lerna notice cli v3.22.1
lerna info versioning independent
lerna info Bootstrapping 15 packages
lerna info Installing external dependencies
lerna ERR! npm install exited 1 in '@joplin/server'
lerna ERR! npm install stdout:

> sqlite3@4.1.0 install /home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

make: Entering directory '/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/build'
  ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlite-autoconf-3280000/sqlite3.c
make: Leaving directory '/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/build'
Failed to execute '/usr/local/lib/nodejs/node-v14.15.4-linux-x64/bin/node /usr/local/lib/nodejs/node-v14.15.4-linux-x64/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64 --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83' (1)

lerna ERR! npm install stderr:
node-pre-gyp WARN Using request for node-pre-gyp https download 
node-pre-gyp WARN Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v4.1.0/node-v83-linux-x64.tar.gz 
node-pre-gyp WARN Pre-built binaries not found for sqlite3@4.1.0 and node@14.15.4 (node-v83 ABI, glibc) (falling back to source compile with node-gyp) 
/bin/sh: 1: python: not found
make: *** [deps/action_before_build.target.mk:13: Release/obj/gen/sqlite-autoconf-3280000/sqlite3.c] Error 127
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/nodejs/node-v14.15.4-linux-x64/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
gyp ERR! System Linux 5.8.0-38-generic
gyp ERR! command "/usr/local/lib/nodejs/node-v14.15.4-linux-x64/bin/node" "/usr/local/lib/nodejs/node-v14.15.4-linux-x64/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64" "--napi_version=7" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v83"
gyp ERR! cwd /home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3
gyp ERR! node -v v14.15.4
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok 
node-pre-gyp ERR! build error 
node-pre-gyp ERR! stack Error: Failed to execute '/usr/local/lib/nodejs/node-v14.15.4-linux-x64/bin/node /usr/local/lib/nodejs/node-v14.15.4-linux-x64/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3/lib/binding/node-v83-linux-x64 --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/home/elsiehupp/Repositories/joplin/packages/server/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:1048:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)
node-pre-gyp ERR! System Linux 5.8.0-38-generic
node-pre-gyp ERR! command "/usr/local/lib/nodejs/node-v14.15.4-linux-x64/bin/node" "/home/elsiehupp/Repositories/joplin/packages/server/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /home/elsiehupp/Repositories/joplin/packages/server/node_modules/sqlite3
node-pre-gyp ERR! node -v v14.15.4
node-pre-gyp ERR! node-pre-gyp -v v0.11.0
node-pre-gyp ERR! not ok 
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.3 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.2.1 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.2.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! sqlite3@4.1.0 install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the sqlite3@4.1.0 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-19T00_31_25_145Z-debug.log

lerna ERR! npm install exited 1 in '@joplin/server'
lerna WARN complete Waiting for 7 child processes to exit. CTRL-C to exit immediately.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! root@ bootstrap: `lerna bootstrap --no-ci`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the root@ bootstrap script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-19T00_31_25_223Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! root@ postinstall: `npm run bootstrap --no-ci && npm run build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the root@ postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-19T00_31_25_262Z-debug.log

And, yes, npm no longer shows anything installed globally:

$ npm list -g --depth=0
/usr/local/lib/nodejs/node-v14.15.4-linux-x64/lib
└── npm@6.14.10

Again, the Python dependency is an upstream issue with node-sqlite3, and manually installing Python fixes it:

$ sudo apt install python
$ npm i

> root@ postinstall /home/elsiehupp/Repositories/joplin
> npm run bootstrap --no-ci && npm run build

> root@ bootstrap /home/elsiehupp/Repositories/joplin
> lerna bootstrap --no-ci

lerna notice cli v3.22.1
lerna info versioning independent
lerna info Bootstrapping 15 packages
lerna info Installing external dependencies
lerna info Symlinking packages and binaries
lerna info lifecycle @joplin/app-clipper@1.0.8~postinstall: @joplin/app-clipper@1.0.8
lerna info lifecycle @joplin/app-desktop@1.7.0~postinstall: @joplin/app-desktop@1.7.0
lerna info lifecycle @joplin/app-mobile@0.8.9~postinstall: @joplin/app-mobile@0.8.9

> @joplin/app-clipper@1.0.8 postinstall /home/elsiehupp/Repositories/joplin/packages/app-clipper
> cd popup && npm install

> core-js@2.6.11 postinstall /home/elsiehupp/Repositories/joplin/packages/app-clipper/popup/node_modules/babel-runtime/node_modules/core-js
> node -e "try{require('./postinstall')}catch(e){}"

> core-js@3.6.4 postinstall /home/elsiehupp/Repositories/joplin/packages/app-clipper/popup/node_modules/core-js
> node -e "try{require('./postinstall')}catch(e){}"

> core-js-pure@3.6.5 postinstall /home/elsiehupp/Repositories/joplin/packages/app-clipper/popup/node_modules/core-js-pure
> node -e "try{require('./postinstall')}catch(e){}"

> joplin-webclipper-popup@0.1.0 postinstall /home/elsiehupp/Repositories/joplin/packages/app-clipper/popup
> node postinstall.js && npm run build

> joplin-webclipper-popup@0.1.0 build /home/elsiehupp/Repositories/joplin/packages/app-clipper/popup
> node scripts/build.js SKIP_PREFLIGHT_CHECK

Creating an optimized production build...
Browserslist: caniuse-lite is outdated. Please run the following command: `npm update`
Compiled successfully.

File sizes after gzip:

  48.12 KB  build/static/js/2.chunk.js
  7.52 KB   build/static/js/main.chunk.js
  775 B     build/static/js/runtime-main.js
  746 B     build/static/css/main.chunk.css

The project was built assuming it is hosted at ./.
You can control this with the homepage field in your package.json.

The build folder is ready to be deployed.

Find out more about deployment here:

  bit.ly/CRA-deploy

npm WARN The package fs-extra is included as both a dev and production dependency.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.11 (node_modules/webpack-dev-server/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.11: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.11 (node_modules/watchpack/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.11: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.12 (node_modules/react-scripts/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.12: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.11 (node_modules/jest-haste-map/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.11: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.2 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 1852 packages from 869 contributors and audited 2131 packages in 39.117s

60 packages are looking for funding
  run `npm fund` for details

found 8831 vulnerabilities (8778 low, 33 moderate, 20 high)
  run `npm audit fix` to fix them, or `npm audit` for details

> @joplin/app-desktop@1.7.0 postinstall /home/elsiehupp/Repositories/joplin/packages/app-desktop
> npm run build

> @joplin/app-desktop@1.7.0 build /home/elsiehupp/Repositories/joplin/packages/app-desktop
> gulp build

/home/elsiehupp/Repositories/joplin
[19:39:31] Using gulpfile ~/Repositories/joplin/packages/app-desktop/gulpfile.js
[19:39:31] Starting 'build'...
[19:39:31] Starting 'compileScripts'...
[19:39:31] Starting 'compilePackageInfo'...
[19:39:31] Starting 'copyPluginAssets'...
[19:39:31] Starting 'copyTinyMceLangs'...
[19:39:31] Starting 'updateIgnoredTypeScriptBuild'...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/ClipperConfigScreen.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/DialogButtonRow.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/EncryptionConfigScreen.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/ExtensionBadge.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/HelpButton.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/IconButton.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/ImportScreen.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/ItemList.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/Navigator.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/NotePropertiesDialog.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/NoteRevisionViewer.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/NoteSearchBar.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/NoteStatusBar.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/PromptDialog.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/TagItem.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/ToolbarSpace.jsx...
Compiling /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/VerticalResizer.jsx...
Copying to /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../gui/note-viewer/pluginAssets
Copying /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../../../Assets/TinyMCE/langs => /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../node_modules/tinymce/langs
[19:39:37] Finished 'compileScripts' after 6.44 s
[19:39:37] Finished 'compilePackageInfo' after 6.44 s
[19:39:38] Finished 'updateIgnoredTypeScriptBuild' after 6.46 s
Copying to /home/elsiehupp/Repositories/joplin/packages/app-desktop/tools/../pluginAssets
[19:39:38] Finished 'copyTinyMceLangs' after 6.48 s
[19:39:38] Finished 'copyPluginAssets' after 6.49 s
[19:39:38] Finished 'build' after 6.49 s

> @joplin/app-mobile@0.8.9 postinstall /home/elsiehupp/Repositories/joplin/packages/app-mobile
> jetify && npm run build

sh: 1: jetify: not found
lerna info lifecycle @joplin/app-mobile@0.8.9~postinstall: Failed to exec postinstall script
lerna ERR! lifecycle "postinstall" errored in "@joplin/app-mobile", exiting NaN
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! root@ bootstrap: `lerna bootstrap --no-ci`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the root@ bootstrap script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-19T00_39_38_071Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! root@ postinstall: `npm run bootstrap --no-ci && npm run build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the root@ postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/elsiehupp/.npm/_logs/2021-01-19T00_39_38_106Z-debug.log

Note that the node-pre-gyp error is nowhere to be found.

Let's what's next:

> @joplin/app-mobile@0.8.9 postinstall /home/elsiehupp/Repositories/joplin/packages/app-mobile
> jetify && npm run build

sh: 1: jetify: not found

Hmmm! I wonder if globally installing jetify (or jetifier) will fix this! Shall I continue?

elsiehupp commented 3 years ago

I did some more digging in the build instructions, and there is, in fact, a section workarounds for macOS and Linux in build_troubleshooting.md. So for the upstream Python dependency issue on ubuntu-x64 in particular, there is precedent for including workarounds in the build instructions. (I should also just open an issue with mapbox about it.) As for npm not being able to find certain packages, you already have a recommendation to globally install node-gyp, so globally installing certain npm packages on macOS or Linux is also not unprecedented.

EDIT: Oh, look, the node-sqlite3 binaries returning a 403 error is already a known issue on their GitHub page: https://github.com/mapbox/node-sqlite3/issues/1389

elsiehupp commented 3 years ago

Well, I've managed to fix both the Linux dependency issues and the macOS build issues.

The Linux dependency issues seem to be just because npm is extraordinarily finicky and legitimately buggy, so under some circumstances it can npm install can remove dependencies rather than install them. Or something. The issue went away after I ran npm install directly in joplin/packages/app-mobile and then ran it again in joplin. Somewhat strangely, even after wiping my cloned copy of the repository and removing and reinstalling node, I couldn't get the problem to reappear.

I can't get the 403 error to reappear either, for what it's worth, so it seems like the other error might have generated an inaccurate download URL for node-pre-gyp.

The macOS build error seems to have been due to the fact that I was using Node 15 rather than Node 14. All of the build errors went away when I manually downgraded. I can open an issue suggesting that the Node version be specified in BUILD.md

elsiehupp commented 3 years ago

For newer variations of the "npm removes depencies" bug, here's a search on the new npm repository:

https://github.com/npm/cli/issues?q=is%3Aissue+is%3Aopen+removes+dependencies

The reason I linked the particular issue I did was that it had the most information and corroboration. Also incidentally people kept commenting that the bug was still present even well after the original author closed the issue.