search

kriszyp / lmdb-js

Simple, efficient, ultra-fast, scalable data store wrapper for LMDB
Other
505 stars 41 forks source link

Error: No native build was found on Docker node:16-alpine environment #82

Closed krnlde closed 2 years ago

krnlde commented 3 years ago

lmdb-store, which is used by parcel bundler v2.0.0.rc.0 doesn't compile on Docker FROM node:16-alpine.

Here's its output.

 > [8/8] RUN node demo.js:
#12 0.574 /app/node_modules/node-gyp-build/index.js:59
#12 0.574   throw new Error('No native build was found for ' + target + '\n    loaded from: ' + dir + '\n')
#12 0.574   ^
#12 0.574
#12 0.574 Error: No native build was found for platform=linux arch=x64 runtime=node abi=93 uv=1 libc=musl node=16.6.0
#12 0.574     loaded from: /app/node_modules/lmdb-store
#12 0.574
#12 0.574     at Function.load.path (/app/node_modules/node-gyp-build/index.js:59:9)
#12 0.574     at load (/app/node_modules/node-gyp-build/index.js:21:30)
#12 0.574     at Object.<anonymous> (/app/node_modules/lmdb-store/index.js:7:49)
#12 0.574     at Module._compile (node:internal/modules/cjs/loader:1101:14)
#12 0.574     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
#12 0.574     at Module.load (node:internal/modules/cjs/loader:981:32)
#12 0.574     at Function.Module._load (node:internal/modules/cjs/loader:822:12)
#12 0.574     at Module.require (node:internal/modules/cjs/loader:1005:19)
#12 0.574     at require (node:internal/modules/cjs/helpers:94:18)
#12 0.574     at Object.<anonymous> (/app/demo.js:1:19)
------
executor failed running [/bin/sh -c node demo.js]: exit code: 1

X-Ref: https://github.com/parcel-bundler/parcel/issues/6680

To reproduce: Create a folder to your liking and add

package.json

{
  "name": "lmdb-demo",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "lmdb-store": "^1.6.4"
  }
}

Dockerfile

FROM node:16-alpine

RUN mkdir /app

COPY package.json /app
COPY yarn.lock /app/

WORKDIR /app

COPY demo.js /app/demo.js
RUN yarn install --ignore-scripts --frozen-lockfile

RUN node demo.js

demo.js

const lmdbStore = require('lmdb-store');

console.log(lmdbStore);

and finally run docker build .

kriszyp commented 3 years ago

I believe this is because you are using --ignore-scripts which specifically tells yarn not to run the script that compiles/installs lmdb-store. So if you are explicitly telling it not to build/install lmdb-store, then lmdb-store can't run.

krnlde commented 3 years ago

Sounds reasonable. Unfortunately, in CI environments you usually want to ignore scripts. Maybe we need to find a way how others are doing it? Like sharp for example, which compiles natively too.

Edit: But you're right, without that flag it builds.

kriszyp commented 3 years ago

It looks like sharp has an install script as well, and behaves exactly the same. If I attempt to install sharp with --ignore-scripts I get:

Cannot find module '../build/Release/sharp.node'

Which is then followed up by a message explicitly instructing me to not use ignore-scripts (or disable the ignoring):

- Remove the "node_modules/sharp" directory then run
  "npm install --ignore-scripts=false --verbose sharp" and look for errors

I do provide pre-compiled binaries for Windows and MacOS, which means it will work on those platforms without any install script. However, I do that by running prebuilds on my windows and mac computers, but trying to provide binaries for all the unix flavors out there would require some CI setup beyond what I have managed to figure out so far. (I did attempt to try out using Travis for prebuilding https://github.com/DoctorEvidence/lmdb-store/commit/ab756d45e0e6b92542d9d8885ef783215c5368a6, but a couple weeks later and still Travis still hasn't built anything https://travis-ci.org/github/DoctorEvidence/lmdb-store)

krnlde commented 3 years ago

Understandably. Alright, so the suggestion is to remove --ignore-scripts flag. Then we need to migrate the CI/CD Pipeline accordingly, but that seems a necessary and reasonable step.

Related: https://github.com/yarnpkg/yarn/issues/4100

willstott101 commented 3 years ago

Compiling takes quite a long time for me, and python needs to be available to do it - quite considerably increasing docker CI run time when this package is involved. Perhaps since this is hosted on gitlab it might be worth giving gitlab actions a go? travis-ci.org has gone btw, and travis-ci.com is the new domain

kriszyp commented 3 years ago

Would it be helpful if I started including precompiled linux binaries? I have a box running Mint, and I could probably include linux-x64 binaries. Not sure the full extent of how widely that can be used on different *nix servers/CI out there though.

kriszyp commented 3 years ago

Ok, I included binaries for linux for node v16 and v14 for x64 and arm64, in the latest, v1.6.6. Let me know if that helps (or causes any issues).

kriszyp commented 2 years ago

I know this issue is closed, but I am not sure that this was fully solved; I don't think standard linux binaries themselves are sufficient here, since I think we also need binaries compiled with musl libc for alpine, IIUC. In the latest v2.0 beta (beta3), I included x86 musl binaries, so if anyone comes across this, and is running lmdb on alpine, I would be curious if these prebuilt binaries are working there.

gogoout commented 2 years ago

Hi, I'm testing lmdb on our ci, seems still failed as no prebuilt binary found. My base image is node:14.18.0-alpine

> lmdb@2.0.0-beta3 install /app/node_modules/lmdb
> node-gyp-build
gyp ERR! find Python 
gyp ERR! find Python Python is not set from command line or npm configuration
gyp ERR! find Python Python is not set from environment variable PYTHON
gyp ERR! find Python checking if "python" can be used
gyp ERR! find Python - "python" is not in PATH or produced an error
gyp ERR! find Python checking if "python2" can be used
gyp ERR! find Python - "python2" is not in PATH or produced an error
gyp ERR! find Python checking if "python3" can be used
gyp ERR! find Python - "python3" is not in PATH or produced an error
gyp ERR! find Python 
gyp ERR! find Python **********************************************************
gyp ERR! find Python You need to install the latest version of Python.
gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
gyp ERR! find Python you can try one of the following options:
gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
gyp ERR! find Python   (accepted by both node-gyp and npm)
gyp ERR! find Python - Set the environment variable PYTHON
gyp ERR! find Python - Set the npm configuration variable python:
gyp ERR! find Python   npm config set python "/path/to/pythonexecutable"
gyp ERR! find Python For more information consult the documentation at:
gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
gyp ERR! find Python **********************************************************
gyp ERR! find Python 
gyp ERR! configure error 
gyp ERR! stack Error: Could not find any Python installation to use
gyp ERR! stack     at PythonFinder.fail (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:307:47)
gyp ERR! stack     at PythonFinder.runChecks (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:136:21)
gyp ERR! stack     at PythonFinder.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:179:16)
gyp ERR! stack     at PythonFinder.execFileCallback (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:271:16)
gyp ERR! stack     at exithandler (child_process.js:390:5)
gyp ERR! stack     at ChildProcess.errorhandler (child_process.js:402:5)
gyp ERR! stack     at ChildProcess.emit (events.js:400:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:280:12)
gyp ERR! stack     at onErrorNT (internal/child_process.js:469:16)
gyp ERR! stack     at processTicksAndRejections (internal/process/task_queues.js:82:21)
gyp ERR! System Linux 5.4.0-1056-gcp
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /app/node_modules/lmdb
gyp ERR! node -v v14.18.0
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! lmdb@2.0.0-beta3 install: `node-gyp-build`
npm ERR! Exit status 1

Do you consider reopen this ticket or shall I open a new one?

kriszyp commented 2 years ago

If you run require('node-gyp-build')('/app/node_modules/lmdb') in node, does it give a more complete error message for the configuration for the prebuild it is looking for? I would expect that lmdb/prebuilds/linux-x64/node.abi83.musl.node should be the correct prebuild for your system (assuming npm didn't delete that directory when it failed, I think you could use --ignore-scripts to ensure it completes without any compilation attempt/failure). Do you know if that binary is there? You don't happen to be on arm64, are you?

gogoout commented 2 years ago

I'm on new m1 which sadly is arm64 😂 But I hack a bit and get the log from ci

Step 11/13 : RUN ls /app/node_modules/lmdb/prebuilds/linux-x64
 ---> Running in 7c8d6f5d1004
electron.abi98.node
node.abi102.musl.node
node.abi102.node
node.abi72.node
node.abi83.musl.node
node.abi83.node
node.abi88.node
node.abi93.musl.node
node.abi93.node
Removing intermediate container 7c8d6f5d1004
 ---> d0025eb36f55
Step 12/13 : COPY . /app
 ---> da2f0aa801dd
Step 13/13 : RUN node ./remove-after-test.js
 ---> Running in 650a9a927b4b
internal/modules/cjs/loader.js:1144
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^
Error: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /app/node_modules/lmdb/prebuilds/linux-x64/node.abi83.musl.node)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1144:18)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:93:18)
    at load (/app/node_modules/node-gyp-build/index.js:21:10)
    at Object.<anonymous> (/app/remove-after-test.js:1:26)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32) {
  code: 'ERR_DLOPEN_FAILED'
}
kriszyp commented 2 years ago

Alpine on an M1. Nice. I'll add a linux arm64 musl build, and see if that helps.

gogoout commented 2 years ago

Oh I might be misunderstood, the above log come from CI which is musl-x86, the m1 stuff is just my local enviroment.

gogoout commented 2 years ago

And also the workaround to install lmdb is just add a line to the alpine image, it seems successfully built

apk add --no-cache python3 py3-pip build-base
kriszyp commented 2 years ago

Oh, ok, I see. So Error loading shared library ld-linux-x86-64.so.2 is the actual error, hmm...

kriszyp commented 2 years ago

@gogoout I published a v2.0.0-beta4 with the musl binaries built on the alpine container you referenced. Can you try that and see if works (without python/building)?

gogoout commented 2 years ago

Yep, I can confirm it installed successfully on my ci.

gogoout commented 2 years ago

Seems on lmdb@2.1.0-beta2 the binary is missing again, seems on 2.1.0-beta1 it was still there. https://www.jsdelivr.com/package/npm/lmdb?path=prebuilds%2Flinux-x64

kriszyp commented 2 years ago

Yes, there was a build failure on alpine while I was building the last beta. The issue that caused it is fixed and will be available in next release. Thanks for pointing this out!

coderste commented 2 years ago

I'm still getting this error when trying to run parceljs inside a docker container.

Error:

client  | > parcel ui/index.html -p 8080 --hmr-port 33333
client  |
client  | Error: No native build was found for platform=linux arch=x64 runtime=node abi=93 uv=1 libc=musl node=16.15.0
client  |     loaded from: /client/node_modules/lmdb and package: lmdb-linux-x64
client  |
client  |     at Function.load.path (/client/node_modules/lmdb/node_modules/node-gyp-build-optional-packages/index.js:65:9)
client  |     at load (/client/node_modules/lmdb/node_modules/node-gyp-build-optional-packages/index.js:21:30)
client  |     at Object.<anonymous> (/client/node_modules/lmdb/dist/index.cjs:47:64)
client  |     at Module._compile (/client/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
client  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
client  |     at Module.load (node:internal/modules/cjs/loader:981:32)
client  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12)
client  |     at Module.require (node:internal/modules/cjs/loader:1005:19)
client  |     at require (/client/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
client  |     at _lmdb (/client/node_modules/@parcel/cache/lib/LMDBCache.js:51:39)
client exited with code 1

Dockerfile

FROM node:16.15.0-alpine

# Working directory
WORKDIR /client

RUN apk update && apk add --no-cache nodejs npm python3 py3-pip build-base

COPY ./package.json .
COPY ./package-lock.json .

RUN npm install --slient

ENV PATH /client/node_modules/.bin:$PATH

COPY . .

EXPOSE 33333

package.json

{
  "source": "ui/index.html",
  "scripts": {
    "start": "parcel ui/index.html -p 8080 --hmr-port 33333",
    "build": "parcel build"
  },
  "dependencies": {
    "react": "^18.1.0",
    "react-dom": "^18.1.0"
  },
  "devDependencies": {
    "@types/react": "^18.0.9",
    "@types/react-dom": "^18.0.5",
    "@typescript-eslint/eslint-plugin": "^5.26.0",
    "eslint-plugin-react-hooks": "^4.5.0",
    "parcel": "^2.6.0",
    "process": "^0.11.10",
    "typescript": "^4.7.2"
  },
}

I'm using a node-alpine image and I've added .parcel-cache to the .dockerignore file

kriszyp commented 2 years ago

Do you see any differences with this attempt to reproduce this issue? https://github.com/DoctorEvidence/lmdb-js/runs/6604028286?check_suite_focus=true (I think this is the same docker container and seems to install with no compilation and properly find the binary)

coderste commented 2 years ago

Unfortunately not I've changed my Dockerfile to the following

FROM node:16.15.0-alpine

# Working directory
WORKDIR /client

RUN apk update && apk add --no-cache nodejs npm python3 py3-pip build-base

# COPY ./package.json .
# COPY ./package-lock.json .

RUN npm install lmdb@^2.4.4

COPY . .

EXPOSE 33333

CMD [ "node", "-e", "console.log(require('lmdb'))" ]

I get a slightly different error now compared to the first one but still related to the lmdb-linux-x64

client  | /client/node_modules/node-gyp-build-optional-packages/index.js:64
client  |   throw new Error('No native build was found for ' + target + '\n    loaded from: ' + dir + ' and package: ' + platformPackage + '\n')
client  |   ^
client  |
client  | Error: No native build was found for platform=linux arch=x64 runtime=node abi=93 uv=1 libc=musl node=16.15.0
client  |     loaded from: /client/node_modules/lmdb and package: @lmdb/lmdb-linux-x64
client  |
client  |     at Function.load.path (/client/node_modules/node-gyp-build-optional-packages/index.js:64:9)
client  |     at Object.load [as default] (/client/node_modules/node-gyp-build-optional-packages/index.js:20:30)
client  |     at Object.<anonymous> (/client/node_modules/lmdb/dist/index.cjs:47:47)
client  |     at Module._compile (node:internal/modules/cjs/loader:1105:14)
client  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
client  |     at Module.load (node:internal/modules/cjs/loader:981:32)
client  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12)
client  |     at Module.require (node:internal/modules/cjs/loader:1005:19)
client  |     at require (node:internal/modules/cjs/helpers:102:18)
client  |     at [eval]:1:13
client exited with code 1
coderste commented 2 years ago

You know what I think this may be because parcel hasn't updated their cache package to lmdb > 2.4.0 they're still installing 2.3.10 which didn't have the dist binary so I can open a PR over there for that

kriszyp commented 2 years ago

v2.3.10 should have binaries as well. They should be in node_modules/lmdb-linux-x64/node.abi93.musl.node. In v2.4, the location of the binaries has moved (to namespaced package @lmdb/lmdb-linux-x64), but that other than that, shouldn't be any changes in binary availabilities between these releases.

kriszyp commented 2 years ago

Here is another run that matches your exact docker/node version and the package update prior to running npm install: https://github.com/DoctorEvidence/lmdb-js/runs/6608682734?check_suite_focus=true Not sure how this substantively differs from how you are running it, it looks basically the same to me?

It might be interesting to ls /client/node_modules/@lmdb/lmdb-linux-x64 (or without the @lmdb for v2.3) to see if the binaries are indeed installed and present (I don't know if something is going wrong with the install or the runtime lookup of the binary file location)

coderste commented 2 years ago

It seems my main issue comes from @parcel/cache inside the node_modules 

client  | Error: No native build was found for platform=linux arch=x64 runtime=node abi=93 uv=1 libc=musl node=16.15.0
client  |     loaded from: /client/node_modules/@parcel/cache/node_modules/lmdb and package: lmdb-linux-x64

As you can see inside the @parcel/cache it can't find the package inside the lmdb which is inside that node_modules running an ls there shows this

 frontend   master ●  ls node_modules/@parcel/cache/node_modules/lmdb
LICENSE          binding.gyp      deps.ts          index.d.ts       level.js         open.js          rollup.config.js write.js
README.md        caching.js       dict             index.js         mod.ts           package.json     src
SECURITY.md      dependencies     dist             keys.js          native.js        read.js          util

Seems like that binary isn't there

kriszyp commented 2 years ago

Well, node_modules/lmdb should have the javascript (and it does), but node_modules/lmdb-linux-x64 should have the binaries (which is now /client/node_modules/@parcel/cache/node_modules/lmdb-linux-x64, maybe this moved because you are installing both versions?)

coderste commented 2 years ago

Hmmm yeah it seems like @parcel/cache/node_modules is not installing the lmdb-linux-x64 binary even though it is request by the package-lock.json file 😕 the only binaries I have inside that @parcel/cache/node_modules are node-gyp-build-***

coderste commented 2 years ago

I think I figured out the one issue and that was I had parcel installed as a devDependancy rather than just a dependancy which means it obviously installs different things. That then just lead me to this error:

client  |
client  | Error: Error loading shared library /client/node_modules/lmdb/build/Release/lmdb.node: Exec format error
client  |     at Object.Module._extensions..node (node:internal/modules/cjs/loader:1189:18)
client  |     at Module.load (node:internal/modules/cjs/loader:981:32)
client  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12)
client  |     at Module.require (node:internal/modules/cjs/loader:1005:19)
client  |     at require (/client/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
client  |     at load (/client/node_modules/node-gyp-build-optional-packages/index.js:21:10)
client  |     at Object.<anonymous> (/client/node_modules/lmdb/dist/index.cjs:47:64)
client  |     at Module._compile (/client/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
client  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
client  |     at Module.load (node:internal/modules/cjs/loader:981:32) {
client  |   code: 'ERR_DLOPEN_FAILED'
client  | }
client exited with code 1

Not as sure what that means to be honest but I think my original problem is now solved so for that we can re-close this issue :)

kriszyp commented 2 years ago

parcel installed as a devDependancy rather than just a dependancy

I thought this was pretty standard practice, typically parcel is used for dev/building, not a runtime, so I think this should be fine to do, and should still install the binary package (and I can verify that it is installed locally on my computer when I do npm install parcel --save-dev. Certainly something that could prevent the installation of the binary packages is if somehow npm is using the --optional flag (and if you also disabled scripts, that would prevent building the binaries).

Error: Error loading shared library /client/node_modules/lmdb/build/Release/lmdb.node: Exec format error

In this latest error, it looks like it is not using any prebuilds (still a mystery), and it is compile/building the binary. However, it looks like it is (cross?) compiling to the wrong architecture or format or something?

coderste commented 2 years ago

I thought that also which is why it's starting to baffle me why when it's inside devDependancy it's not installing everything it seems to need. I'm not running with disabled-scripts either or the optional flag.

What I've just tried is sh into the docker container and removing the node_modules and package-lock.json and then just running npm inside the container. Once I did this and ran my start command parcel work just now I'm leaning towards what you said at the end there about some weird cross compiling is happening or something although I'm not sure how that happens. I believe it may be that I have to add some folders to my .dockerignore (possibly?)

coderste commented 2 years ago

Solved this now. The main issue I believe was I need to add node_modules to .dockerignore I also presisted the client/node_modules to the volumes inside the docker-compose.yml this then seemed to solve all the issues I was having with parcel so happy days I suppose. Thanks for all the help in trying to debug this issue @kriszyp

kriszyp commented 2 years ago

Ah, great to hear! (and glad you figured it out because this was getting well beyond my devops/containers knowledge :) )

mountainash commented 2 years ago

Can across this issue when trying to run parcel@2.6.2 on node:18-alpine. After much reading and failed builds, my working local Dockerfile is: 

FROM node:18-alpine
ENV NODE_ENV=development
WORKDIR /usr/src/app
COPY package.json /usr/src/app/
RUN apk add --update --no-cache --virtual .build-deps \
build-base \
python3 && \
npm install && \
apk del .build-deps

Nice and easy to copy and paste if any other Devs find themselves here.