ava test framework with worker threads - Module did not self-register

simplenotezy commented 2 years ago

Possible bug

I've been noticing this is quite a normal error message to receive, but most of the suggestions recommend deleting node_modules, or similar - which is not really relevant in my case, since the error only happens during a Github Actions deployment.

My github actions step looks something like this:

        run: |
          npm ci
          npm run build --if-present
          npm run test

I have tried adding npm install --platform=linux --ignore-scripts=false --arch=x64 sharp after npm ci but it will give me the same error.
I have even tried pushing a build without the package-lock.json to see if that would make a difference during deployment, but to no avail.
I verified node and npm version are same on my local machine as well as on the Github actions runner
Output of npm install --platform=linux --arch=x64 --verbose sharp in the github actions runner: https://gist.github.com/simplenotezy/bb1dc5fa506c8629c03d80703247969e

I am not really sure how to proceed here.

❯ npm ls sharp
project@0.0.1 /Users/mf/Projects/project/backend
└── sharp@0.30.3

❯ npm why sharp
sharp@0.30.3
node_modules/sharp
  sharp@"^0.30.3" from the root project

Not sure if it is related to worker threads (https://sharp.pixelplumbing.com/install#worker-threads). The error does occur when running my tests, which I believe could be multi-threaded (Using Ava 4)

Is this a possible bug in a feature of sharp, unrelated to installation?

[x] Running npm install sharp completes without error.
[x] Running node -e "require('sharp')" completes without error.

Are you using the latest version of sharp?

[x] I am using the latest version of sharp as reported by npm view sharp dist-tags.latest.

What is the output of running `npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp`?

System:
    OS: Linux 5.11 Ubuntu 20.04.4 LTS (Focal Fossa)
    CPU: (2) x64 Intel(R) Xeon(R) Platinum 8171M CPU @ 2.[60](https://github.com/DoubbleDK/backend/runs/5741770630?check_suite_focus=true#step:7:60)GHz
    Memory: 5.74 GB / 6.78 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 16.14.0 - /opt/hostedtoolcache/node/16.14.0/x[64](https://github.com/DoubbleDK/backend/runs/5741770630?check_suite_focus=true#step:7:64)/bin/node
    Yarn: 1.22.17 - /usr/local/bin/yarn
    npm: 8.3.1 - /opt/hostedtoolcache/node/16.14.0/x64/bin/npm
  npmPackages:
    sharp: ^0.30.3 => 0.30.3

What is the expected behaviour?

Should install without errors

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem

This happens during my Github Actions deployment, of a Nest.js framework that uses Sharp.

    steps:
      - uses: actions/checkout@v2
      - name: Use Node.js 16.x
        uses: actions/setup-node@v1
        with:
          node-version: 16.x
      - name: Run tests
        run: |
          npm ci
          npm run build --if-present
          npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp
          npm run test

Please provide sample image(s) that help explain this problem

  Error: 
  Something went wrong installing the "sharp" module

  Module did not self-register: '/home/runner/work/backend/backend/node_modules/sharp/build/Release/sharp-linux-x64.node'.

  Possible solutions:
  - Install with the --verbose flag and look for errors: "npm install --ignore-scripts=false --verbose sharp"
  - Install for the current linux-x64 runtime: "npm install --platform=linux --arch=x64 sharp"
  - Consult the installation documentation: https://sharp.pixelplumbing.com/install

  › - Consult the installation documentation: https://sharp.pixelplumbing.com/install
  › Object.<anonymous> (node_modules/sharp/lib/sharp.js:31:9)
  › Object.<anonymous> (node_modules/sharp/lib/constructor.js:8:1)

lovell commented 2 years ago

Not sure if it is related to worker threads (https://sharp.pixelplumbing.com/install#worker-threads). The error does occur when running my tests, which I believe could be multi-threaded (Using Ava 4)

Did you try ava --no-worker-threads ?

simplenotezy commented 2 years ago

@lovell Thanks for prompt reply. Just realised it might be due to that, so I am about to issue a deploy without worker threads. If this is the case, my next question would be: how can I have ava with multiple worker threads to work with Sharp?

simplenotezy commented 2 years ago

Yes, it actually fixed it.

lovell commented 2 years ago

https://sharp.pixelplumbing.com/install#worker-threads

The main thread must call require('sharp') before worker threads are created to ensure shared libraries remain loaded in memory until after all threads are complete.

You'll need to work out how to inject your own logic into ava's main thread.

Alternatively, use a musl-based Linux such as Alpine, which sensibly ignores Node.js' attempt to call dlclose - see https://wiki.musl-libc.org/functional-differences-from-glibc.html#Unloading-libraries

lovell commented 2 years ago

I've slightly expanded the existing docs about this via commit https://github.com/lovell/sharp/commit/d0c8e9564163cb8d1e1096c503f0c03d906f2666

simplenotezy commented 2 years ago

Awesome, thanks!

titanism commented 2 years ago

You can also do this as a better workaround (save this as ava.config.js and remove ava config from your package.json):

const isCI = require('is-ci');

module.exports = {
  workerThreads: !isCI
};

lovell commented 2 years ago

Thanks @titanism that's a neat solution.

Given this is primarily due to a feature of glibc-based Linux, a possible alternative might be something like:

const { familySync, GLIBC } = require('detect-libc');

module.exports = {
  workerThreads: familySync() !== GLIBC
};

codetheweb commented 2 years ago

In my case I needed worker threads to use AVA's shared workers.

Just requiring the module in the config file for AVA:

// ava.config.js
require("sharp");
// ...

worked for me.

lovell commented 2 years ago

The forthcoming libvips v8.13.0, which will ship with the forthcoming sharp v0.31.0, should remove the need for all these workarounds - see https://github.com/libvips/libvips/pull/2934

titanism commented 1 year ago

Hi there @lovell per https://github.com/forwardemail/forwardemail.net/commit/5460f9ab585c31c3c8729f81f8bb5f82df514116 we updated to latest version v0.32.1 and the error still persists with worker threads. This issue is still not resolved. Please re-open or update the worker threads docs at https://sharp.pixelplumbing.com/install#worker-threads to suggest the following workaround:

npm install detect-libc

// ava.config.js
const { familySync, GLIBC } = require('detect-libc');

module.exports = {
  // <https://github.com/lovell/sharp/issues/3164>
  workerThreads: familySync() !== GLIBC
};

titanism commented 1 year ago

@lovell we updated the docs for you, please see PR at https://github.com/lovell/sharp/pull/3692

lovell commented 1 year ago

@titanism As of sharp v0.31.0 this should no longer be a problem. Are you able to verify if the removal of this line still allows your tests to pass? https://github.com/forwardemail/forwardemail.net/blob/748b21304ec5224d14ab5b98e7dc0e51ad96b72f/ava.config.js#L10

luixo commented 8 months ago

Hello, I got a similar problem migrating from a single config of vitest to a workspace setup. I'm not sure what exactly happened. Before migration I used vmThreads pooling method, afterwards I had multiple projects, each using the exact same method. Unfortunately I'm not that into what's happening in vitest to tell exactly what's changed between non-workspace and workspace environments.

Adding import("sharp") into general vitest.config.ts (which is in an outer nodejs context, I presume) resolved the problem.