search

QwikDev / qwik

Instant-loading web apps, without effort
https://qwik.dev
MIT License
20.77k stars 1.29k forks source link

[🐞] Error: Could not load the "sharp" module using the linux-x64 runtime #5556

Closed Przemoo16 closed 5 months ago

Przemoo16 commented 10 months ago

Summary

This issue is due to npm not storing optional platform-dependent dependencies in the lockfile (see issue), so that using the lockfile on another platform fails. pnpm doesn't seem to have a problem with it.

Leaving this open for visibility.

Which component is affected?

Qwik Runtime

Describe the bug

After updating Qwik to the latest version, I'm getting the following error when trying to build an app:

error during build:
Error: Could not load the "sharp" module using the linux-x64 runtime
Possible solutions:
- Add platform-specific dependencies:
    npm install --os=linux --cpu=x64 sharp
  or
    npm install --force @img/sharp-linux-x64
- Consult the installation documentation: https://sharp.pixelplumbing.com/install
    at Object.<anonymous> (/home/przemek/dev/personal/project-starter-graphql/frontend/node_modules/@builder.io/qwik-city/node_modules/sharp/lib/sharp.js:85:9)
    at Module._compile (node:internal/modules/cjs/loader:1233:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1287:10)
    at Module.load (node:internal/modules/cjs/loader:1091:32)
    at Module._load (node:internal/modules/cjs/loader:938:12)
    at Module.require (node:internal/modules/cjs/loader:1115:19)
    at require (node:internal/modules/helpers:130:18)
    at Object.<anonymous> (/home/przemek/dev/personal/project-starter-graphql/frontend/node_modules/@builder.io/qwik-city/node_modules/sharp/lib/constructor.js:10:1)
    at Module._compile (node:internal/modules/cjs/loader:1233:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1287:10)
error Command failed with exit code 1.

It could be related to this issue: https://github.com/lovell/sharp/issues/3870

Reproduction

https://github.com/Przemoo16/qwik-bug

Steps to reproduce

  1. yarn create qwik
  2. yarn build

System Info

System:
    OS: Linux 6.2 Ubuntu 22.04.3 LTS 22.04.3 LTS (Jammy Jellyfish)
    CPU: (8) x64 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
    Memory: 26.08 GB / 31.09 GB
    Container: Yes
    Shell: 5.8.1 - /usr/bin/zsh
  Binaries:
    Node: 20.5.1 - ~/.nvm/versions/node/v20.5.1/bin/node
    Yarn: 1.22.21 - ~/.nvm/versions/node/v20.5.1/bin/yarn
    npm: 9.8.0 - ~/.nvm/versions/node/v20.5.1/bin/npm
  Browsers:
    Brave Browser: 119.1.60.118
  npmPackages:
    @builder.io/qwik: ^1.3.0 => 1.3.0 
    @builder.io/qwik-city: ^1.3.0 => 1.3.0 
    undici: ^5.28.2 => 5.28.2 
    vite: ^5.0.7 => 5.0.7

Additional Information

No response

gioboa commented 10 months ago

HI @Przemoo16 is it related to this one https://github.com/lovell/sharp/issues/3870 ? I'm trying to figure out the exact way to solve the issue.

Przemoo16 commented 10 months ago

Hi @gioboa, yes, the issue seems to be the same, however I'm not using Vercel but they mentioned it is only related to the Vercel environment.

wmertens commented 10 months ago

@Przemoo16 can you try adding

  "optionalDependencies": {
    "@img/sharp-linux-x64": "^0.33.0"    
  },

to your package.json?

lovell commented 10 months ago

Yarn: 1.22.21 - ~/.nvm/versions/node/v20.5.1/bin/yarn

https://sharp.pixelplumbing.com/install

# yarn v1 (maintenance mode)
yarn add sharp --ignore-engines

(yarn v1 has been in maintenance mode for almost 4 years, please upgrade.)

Przemoo16 commented 10 months ago

Yarn: 1.22.21 - ~/.nvm/versions/node/v20.5.1/bin/yarn

https://sharp.pixelplumbing.com/install

# yarn v1 (maintenance mode)
yarn add sharp --ignore-engines

(yarn v1 has been in maintenance mode for almost 4 years, please upgrade.)

Thanks for the reply. This seems to solve the issue. However, I don't want to specify the sharp as my app dependency. This is a dependency that comes from the Qwik so I'd expect that the engines are ignored at the Qwik level.

primozs commented 10 months ago

i have the same issue yarn1 and latest pnpm. pnpm sometimes works but rm -rf node_modules, pnpm install, at this moment does not work for me. Ubuntu 22.04.3 LTS node v20.10.0 nvm 0.38.0

primozs commented 10 months ago

for me solution is: "postinstall": "pnpm install sharp"

kokecar11 commented 10 months ago

I have the same error but with pnpm, any solution?

primozs commented 10 months ago

@Przemoo16 can you try adding

  "optionalDependencies": {
    "@img/sharp-linux-x64": "^0.33.0"    
  },

to your package.json?

works

wmertens commented 10 months ago

Next person with issues please try pnpm dedupe, or when using npm, making sure it's npm>v9.6.5

kokecar11 commented 10 months ago

Next person with issues please try pnpm dedupe, or when using npm, making sure it's npm>v9.6.5

I did the pnpm dedupe but it didn't work and I added the above in my package.json

"optionalDependencies": {
    "@img/sharp-darwin-arm64": "^0.33.0"  
  },
wmertens commented 10 months ago

The relevant issue for npm is here: https://github.com/npm/cli/issues/4828 Other package managers might have similar issues.

The list of optional dependencies that sharp needs at least one of is this: https://github.com/lovell/sharp/blob/c5eaeb2ddbef7fa74fb682c8dc774156d084c053/package.json#L143-L163

wmertens commented 10 months ago

@lovell pinging you for visibility but I'm not sure you can do anything on your end? Or do you know of something specific we could do when the starter project is created?

lovell commented 10 months ago

sharp v0.33.0 onwards has intentionally switched to use exactly the same installation approach as esbuild, rollup, swc etc. and take advantage of features made available in all modern JS package managers. This allows people to select their choice of package manager, and in doing so therefore accept any bugs their choice of package manager might contain.

https://sharp.pixelplumbing.com/install

lovell commented 10 months ago

I've attempted to improve the messaging in sharp via commit https://github.com/lovell/sharp/commit/3a0c37569219d10321f559fc63715ef45aca0c0d but the summary here is that npm is currently the wrong choice of package manager if you want multi-platform support.

DercilioFontes commented 10 months ago

npm i --package-lock=false worked for me.

package-lock Default: true Type: Boolean If set to false, then ignore package-lock.json files when installing. This will also prevent writing package-lock.json if save is true.

gioboa commented 10 months ago

@DeVoresyah you have an issue with Cloudflare and sharp, right?

DeVoresyah commented 10 months ago

@DeVoresyah you have an issue with Cloudflare and sharp, right?

yes, here's my detail:

Cloudflare Build Settings

Build command: bun run build
Build output directory: /dist
Build system version: 2 (latest)
Root directory: /
Environment variables: None

Package Manager Used

bun

Error Log

2023-12-15T07:33:21.472815Z Cloning repository...
2023-12-15T07:33:22.736409Z From https://github.com/Cuan-sh/cuan-landing
2023-12-15T07:33:22.737926Z  * branch            f969cdbb95f1e9e6c0a46a15f87589446a83e26b -> FETCH_HEAD
2023-12-15T07:33:22.738284Z 
2023-12-15T07:33:22.884326Z HEAD is now at f969cdb fix: remove sharp
2023-12-15T07:33:22.884966Z 
2023-12-15T07:33:22.98129Z  
2023-12-15T07:33:23.011128Z Success: Finished cloning repository files
2023-12-15T07:33:23.748422Z Detected the following tools from environment: nodejs@16.20.2, bun@1.0.1
2023-12-15T07:33:23.749321Z Installing nodejs 16.20.2
2023-12-15T07:33:23.832509Z nodejs 16.20.2 is already installed
2023-12-15T07:33:24.404676Z Installing project dependencies: bun install --frozen-lockfile
2023-12-15T07:33:24.666856Z bun install v1.0.1 (31aec4eb)
2023-12-15T07:33:35.150443Z  + @img/sharp-linux-x64@0.33.0
2023-12-15T07:33:35.150862Z  + @builder.io/qwik@1.2.17
2023-12-15T07:33:35.151007Z  + @builder.io/qwik-city@1.2.17
2023-12-15T07:33:35.151123Z  + @storybook/addon-essentials@7.5.3
2023-12-15T07:33:35.151234Z  + @storybook/addon-links@7.5.3
2023-12-15T07:33:35.151351Z  + @storybook/blocks@7.5.3
2023-12-15T07:33:35.151464Z  + @storybook/builder-vite@7.5.3
2023-12-15T07:33:35.151572Z  + @storybook/html@7.5.3
2023-12-15T07:33:35.15169Z   + @storybook/html-vite@7.5.3
2023-12-15T07:33:35.151798Z  + @types/eslint@8.44.7
2023-12-15T07:33:35.151905Z  + @types/node@20.9.0
2023-12-15T07:33:35.152019Z  + @typescript-eslint/eslint-plugin@6.11.0
2023-12-15T07:33:35.152147Z  + @typescript-eslint/parser@6.11.0
2023-12-15T07:33:35.152256Z  + autoprefixer@10.4.16
2023-12-15T07:33:35.15237Z   + eslint@8.53.0
2023-12-15T07:33:35.152508Z  + eslint-plugin-qwik@1.2.17
2023-12-15T07:33:35.152648Z  + postcss@8.4.31
2023-12-15T07:33:35.152765Z  + prettier@3.1.0
2023-12-15T07:33:35.152875Z  + prettier-plugin-tailwindcss@0.5.7
2023-12-15T07:33:35.153004Z  + storybook@7.5.3
2023-12-15T07:33:35.153118Z  + storybook-framework-qwik@0.2.4
2023-12-15T07:33:35.153227Z  + tailwindcss@3.3.3
2023-12-15T07:33:35.153367Z  + typescript@5.2.2
2023-12-15T07:33:35.153482Z  + undici@5.27.2
2023-12-15T07:33:35.153606Z  + vite@4.5.0
2023-12-15T07:33:35.153723Z  + vite-tsconfig-paths@4.2.1
2023-12-15T07:33:35.153832Z  + wrangler@3.20.0
2023-12-15T07:33:35.153953Z  + @fontsource-variable/inter@5.0.15
2023-12-15T07:33:35.154068Z  + @qwikest/icons@0.0.13
2023-12-15T07:33:35.154177Z  + remixicon@3.5.0
2023-12-15T07:33:35.154291Z 
2023-12-15T07:33:35.154401Z  1123 packages installed [10.49s]
2023-12-15T07:33:35.179559Z Executing user command: bun run build
2023-12-15T07:33:35.480493Z $ qwik build
2023-12-15T07:33:35.804543Z 
2023-12-15T07:33:35.805033Z       ............
2023-12-15T07:33:35.805263Z     .::: :--------:.
2023-12-15T07:33:35.805479Z    .::::  .:-------:.
2023-12-15T07:33:35.805636Z   .:::::.   .:-------.
2023-12-15T07:33:35.80584Z    ::::::.     .:------.
2023-12-15T07:33:35.805988Z  ::::::.        :-----:
2023-12-15T07:33:35.806195Z  ::::::.       .:-----.
2023-12-15T07:33:35.806326Z   :::::::.     .-----.
2023-12-15T07:33:35.806449Z    ::::::::..   ---:.
2023-12-15T07:33:35.806651Z     .:::::::::. :-:.
2023-12-15T07:33:35.806786Z      ..::::::::::::
2023-12-15T07:33:35.806896Z              ...::::
2023-12-15T07:33:35.807173Z      
2023-12-15T07:33:35.807349Z 
2023-12-15T07:33:35.8075Z   
2023-12-15T07:33:35.807627Z bun run build.types
2023-12-15T07:33:35.807756Z bun run build.client
2023-12-15T07:33:35.807883Z bun run build.server
2023-12-15T07:33:35.808001Z bun run lint
2023-12-15T07:33:35.808135Z 
2023-12-15T07:33:35.817122Z $ tsc --incremental --noEmit --pretty
2023-12-15T07:33:35.832806Z $ vite build
2023-12-15T07:33:36.708088Z error during build:
2023-12-15T07:33:36.708443Z Error: 
2023-12-15T07:33:36.708578Z Something went wrong installing the "sharp" module
2023-12-15T07:33:36.708735Z 
2023-12-15T07:33:36.708846Z Cannot find module '../build/Release/sharp-linux-x64.node'
2023-12-15T07:33:36.708954Z Require stack:
2023-12-15T07:33:36.709062Z - /opt/buildhome/repo/node_modules/sharp/lib/sharp.js
2023-12-15T07:33:36.70917Z  - /opt/buildhome/repo/node_modules/sharp/lib/constructor.js
2023-12-15T07:33:36.709277Z - /opt/buildhome/repo/node_modules/sharp/lib/index.js
2023-12-15T07:33:36.709386Z 
2023-12-15T07:33:36.709503Z Possible solutions:
2023-12-15T07:33:36.70961Z  - Install with verbose logging and look for errors: "npm install --ignore-scripts=false --foreground-scripts --verbose sharp"
2023-12-15T07:33:36.709732Z - Install for the current linux-x64 runtime: "npm install --platform=linux --arch=x64 sharp"
2023-12-15T07:33:36.709839Z - Consult the installation documentation: https://sharp.pixelplumbing.com/install
2023-12-15T07:33:36.709951Z     at Object.<anonymous> (/opt/buildhome/repo/node_modules/sharp/lib/sharp.js:37:9)
2023-12-15T07:33:36.710058Z     at Module._compile (node:internal/modules/cjs/loader:1198:14)
2023-12-15T07:33:36.710172Z     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)
2023-12-15T07:33:36.710286Z     at Module.load (node:internal/modules/cjs/loader:1076:32)
2023-12-15T07:33:36.710398Z     at Function.Module._load (node:internal/modules/cjs/loader:911:12)
2023-12-15T07:33:36.710516Z     at Module.require (node:internal/modules/cjs/loader:1100:19)
2023-12-15T07:33:36.710632Z     at require (node:internal/modules/cjs/helpers:119:18)
2023-12-15T07:33:36.71075Z      at Object.<anonymous> (/opt/buildhome/repo/node_modules/sharp/lib/constructor.js:11:1)
2023-12-15T07:33:36.713258Z     at Module._compile (node:internal/modules/cjs/loader:1198:14)
2023-12-15T07:33:36.713453Z     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)
2023-12-15T07:33:36.734319Z error: script "build.client" exited with code 1 (SIGHUP)
2023-12-15T07:33:36.742193Z error: script "build" exited with code 1 (SIGHUP)
2023-12-15T07:33:36.744367Z Failed: Error while executing user command. Exited with error code: 1
2023-12-15T07:33:39.331625Z Failed: build command exited with code: 1
2023-12-15T07:33:40.213682Z Failed: error occurred while running build command
DeVoresyah commented 10 months ago

hi @gioboa, for my case I think it's my mistake because not upgrading all qwik dependencies to the latest. I found the root cause when opening the issue in sharp repo (https://github.com/lovell/sharp/issues/3896).

So, I tried to upgrade to qwik latest wich is 1.3.0 and it's solved the issue

fiyazbinhasan commented 10 months ago

facing the same issue on cloudflare pages build pipeline

PatrickJS commented 10 months ago

I added the optional deps to my package.json. I'm using latest npm. works on my machine now both mac/linux packages are downloaded and in github actions I still have npm ci and it uses the correct optional package

  "optionalDependencies": {
    "@img/sharp-linux-x64": "^0.33.0",
    "@rollup/rollup-linux-x64-gnu": "^4.9.0"
  }
xyzwps commented 10 months ago

On macos 14.1.1, I run pnpm install --force @img/sharp-darwin-x64 and it works.

fprl commented 9 months ago

Not ideal but fixed adding "postinstall": "pnpm install --force @img/sharp-darwin-x64" to package.json (for cloudflare).

abdhashem commented 9 months ago

For me, using Mac M1, adding the following to package.json:

"optionalDependencies": {
    "@img/sharp-darwin-arm64": "^0.33.1"
  }

Then, pnpm dedupe solved the problem.

wmertens commented 9 months ago

Please all note that sharp is only working on "node": "^18.17.0 || ^20.3.0 || >=21.0.0", which means that its optional binary dependencies won't be installed on other node versions. Node 16 is not an option.

https://github.com/lovell/sharp/blob/a77ac6ae259908a989d6c73e70f19d6994b7a237/package.json#L189

nelsonprsousa commented 9 months ago

btw, I believe yarn v1 isn't compatible with sharp module. We had to upgrade to yarn v4 (I guess yarn v2+ would work as well).

lovell commented 9 months ago

@nelsonprsousa https://github.com/BuilderIO/qwik/issues/5556#issuecomment-1849900140

lovell commented 9 months ago

@wmertens Thanks, you've inspired me to add a runtime version check via commit https://github.com/lovell/sharp/commit/45e8071599772f320d0d8314fedf0b13b0592d28

nelsonprsousa commented 9 months ago

@nelsonprsousa #5556 (comment)

We tried with that as well. I believe the problem was with the Dockerfile using yarn v1.

But yeah, no longer using v1. No point really in using v1 in 2024

jafsparrow commented 9 months ago

This 'sharp' issue is stopping me from deploying to netlify....! cannot find a solution online.. I wonder why the framework authors do not add steps to get it right.???

tleperou commented 9 months ago

When built on Cloudflare, setting up node to 18+ will resolve the error.

I added a file .node-version at the root, 

18

to instruct CL to build against that version.

jafsparrow commented 9 months ago

When built on Cloudflare, setting up node to 18+ will resolve the error.

I added a file .node-version at the root,

18

to instruct CL to build against that version.

Thank you.. I will try and will update later

PatrickJS commented 5 months ago

closing since we require node 18 now