Engines file missing when ignoring missing checksum

[spudly]

Bug description

I recently added the PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING environment variable because we use Artifactory, which has its own way of storing checksum files that doesn't match what the Prisma binary downloader expects. After doing so, I had difficulty getting the binaries downloaded and into the right location.

First Issue:

Installing works as expected the first time, but the next time I need to reinstall (or install prisma into a different repo), it sees the binary in ~/.cache/prisma, but never copies the files from cache into node_modules. The only way I've found to get the binaries installed is to run rm -rf ~/.cache/prisma before running npm install.

The problem is here: https://github.com/prisma/prisma/blob/main/packages/fetch-engine/src/download.ts#L278-L291

In that code, it ignores the missing checksum and allows the script to continue, but it never copies the binary file from cache. It just logs the checksum failure and returns.

Second issue

When running prism a generate, I get this error:

ENOENT: no such file or directory, copyfile '<my-package>/node_modules/prisma/libquery_engine-darwin.dylib.node' -> '<my-package>/node_modules/.prisma/client/libquery_engine-darwin.dylib.node'

The file exists, but it's not in the node_modules/prisma folder, where it's trying to copy from. It's in node_modules/@prisma/engines/. I'm not sure where the problem is for this one, or if this is related to the first issue or not.

To work around this, I'm running the following command after installing dependencies:

find node_modules -name '*.node' -exec cp -n "{}" node_modules/prisma/ \;

How to reproduce

1. cd into a project that uses prisma
2. run export PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING=true
3. run rm -rf ~/.cache/prisma (clear cache so existing cache doesn't interfere with these steps)
4. rm -rf node_modules (remove node modules so that it actually does a full reinstall)
5. run npm install (this is the first install, and should add prisma binaries to cache, but it may fail to get checksums for those binaries)
6. To simulate an environment (like mine) where the binaries downloaded but the checksum files didn't, you'll probably have to run something like this: find ~/.cache/prisma -name '*.sha256' -delete
7. rm -rf node_modules
8. npm install (this is the second install, which should use the cached binaries, but it doesn't copy the binaries from cache)

Expected behavior

Binaries should be installed during postinstall regardless of whether the binaries are cached or not

Prisma information

n/a

Environment & setup

• OS: macOs
• Database: postGres
• Node.js version: 16.13.1

Prisma Version

Environment variables loaded from .env
prisma                  : 4.16.1
@prisma/client          : 4.16.1
Current platform        : darwin
Query Engine (Node-API) : libquery-engine b20ead4d3ab9e78ac112966e242ded703f4a052c (at node_modules/@prisma/engines/libquery_engine-darwin.dylib.node)
Migration Engine        : migration-engine-cli b20ead4d3ab9e78ac112966e242ded703f4a052c (at node_modules/@prisma/engines/migration-engine-darwin)
Format Wasm             : @prisma/prisma-fmt-wasm 4.16.0-66.b20ead4d3ab9e78ac112966e242ded703f4a052c
Default Engines Hash    : b20ead4d3ab9e78ac112966e242ded703f4a052c
Studio                  : 0.484.0

[linkvt]

I can confirm this issue, it occured to me when installing dependencies in the hoppscotch/hoppscotch repo where the prisma postinstall tries to generate the schema but doesn't find the engine binaries. The strange thing was that the native engine binaries were not copied correctly but the explicit ones (e.g. debian-openssl-1.1.x) were.

The following environment variables are set on my side:

export PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING=1
export PRISMA_ENGINES_MIRROR=https://artifactory.example.com/prisma-binaries-remote

The workaround was as described above to copy the engine files from the @prisma/engines package to the prisma package.

Detailed Workaround

To not copy all *.node files in the node_modules I did the following (using pnpm):

1. Try installing your dependencies with pnpm i, which should fail
2. cd into the directory containing the package.json using prisma
   • in case of hoppscotch this was cd packages/hoppscotch-backend
3. get the prisma path in the node_modules

   $ PRISMA_PATH=$(pnpm why @prisma/engines --json | jq -r '.[0].dependencies.prisma.path')
   $ echo $PRISMA_PATH
   /path/to/my/repos/hoppscotch/node_modules/.pnpm/prisma@4.16.0/node_modules/prisma

4. get the path to the @prisma/engines package

   $ ENGINES_PATH=$(pnpm why @prisma/engines --json | jq -r '.[0].dependencies.prisma.dependencies."@prisma/engines".path')
   $ echo $ENGINES_PATH
   /path/to/my/repos/hoppscotch/node_modules/.pnpm/@prisma+engines@4.16.0/node_modules/@prisma/engines

5. Copy the .node files from engines to prisma

   $ cp $ENGINES_PATH/*.node $PRISMA_PATH
   $ ls $PRISMA_PATH/*.node
   ...

6. Run pnpm i again to trigger the postinstall and generate the schema

Prisma Version

Using Amazon Linux 2, this is the output of npx prisma --version

$ npx prisma --version
prisma                  : 4.16.0
@prisma/client          : 4.16.0
Current platform        : rhel-openssl-1.0.x
Query Engine (Node-API) : libquery-engine b20ead4d3ab9e78ac112966e242ded703f4a052c (at ../../node_modules/.pnpm/@prisma+engines@4.16.0/node_modules/@prisma/engines/libquery_engine-rhel-openssl-1.0.x.so.node)
Migration Engine        : migration-engine-cli b20ead4d3ab9e78ac112966e242ded703f4a052c (at ../../node_modules/.pnpm/@prisma+engines@4.16.0/node_modules/@prisma/engines/migration-engine-rhel-openssl-1.0.x)
Format Wasm             : @prisma/prisma-fmt-wasm 4.16.0-66.b20ead4d3ab9e78ac112966e242ded703f4a052c
Default Engines Hash    : b20ead4d3ab9e78ac112966e242ded703f4a052c
Studio                  : 0.484.0

[RaHehl]

Here is a possible fix, I just don't know how to get it germerged, unfortunately no one responded to my queries in discord https://github.com/prisma/prisma/pull/23001

[Emulator000]

Encountering the same issue using node:20-slim Docker image for installing Prisma ^5.14.0.

 > [backend 8/8] RUN npx prisma generate:
0.820 Prisma schema loaded from prisma/schema.prisma
1.073 Error: 
1.073 ENOENT: no such file or directory, copyfile '/app/node_modules/prisma/libquery_engine-debian-openssl-3.0.x.so.node' -> '/app/node_modules/.prisma/client/libquery_engine-debian-openssl-3.0.x.so.node'
1.073 
1.073 
------
failed to solve: process "/bin/sh -c npx prisma generate" did not complete successfully: exit code: 

I noticed that in the node_modules/prisma folder, only openssl-1.1.x binaries are available and not 3.0.x one.

If I output OpenSSL version inside the image:

OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)

For the time being I resolved the issue manually downloading the binary from the remote registry, unzipping it and putting it inside the node_modules/prisma/libquery_engine-debian-openssl-3.0.x.so.node and it works.

It's a workaround, any reason why this happening?

[BitPatty]

Any image that comes with openssl3 doesn't appear to work without manually downloading the engine somehow

npm ci with prisma generate does not find the engine:

$ npm ci

> some-project@1.9.4 prepare
> if test "$NODE_ENV" != "production" ; then ts-patch install -s ; fi

added 816 packages in 3m

86 packages are looking for funding
  run `npm fund` for details
$ npm run prisma:generate

> some-project@1.9.4 prisma:generate
> prisma generate

prisma:engines binaries to download libquery-engine, schema-engine +288ms
prisma:get-platform Found distro info:
{
  "targetDistro": "musl",
  "familyDistro": "alpine",
  "originalDistro": "alpine"
} +6ms
prisma:get-platform Don't know any platform-specific paths for "alpine" on x64 (x86_64) +0ms
prisma:get-platform Falling back to "ldconfig" and other generic paths +0ms
prisma:get-platform Found libssl.so file using "ldconfig" or other generic paths: libssl.so.3 +10ms
prisma:get-platform The parsed libssl version is: 3.0.x +0ms
prisma:fetch-engine:download The checksum file: /root/.cache/prisma/master/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/libquery-engine.sha256 is missing but this was ignored as the PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING environment variable is truthy. +2ms
prisma:fetch-engine:download The checksum file: /root/.cache/prisma/master/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/schema-engine.sha256 is missing but this was ignored as the PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING environment variable is truthy. +1ms
prisma:loadEnv project root found at /builds/some-project/package.json +312ms
prisma:getSchema prismaConfig {
  "packagePath": "/builds/some-project/package.json"
} +3ms
prisma:tryLoadEnv Environment variables not found at null +0ms
prisma:tryLoadEnv Environment variables not found at undefined +0ms
prisma:tryLoadEnv No Environment variables loaded +0ms
prisma:getSchema prismaConfig {
  "packagePath": "/builds/some-project/package.json"
} +1ms
prisma:getSchema Checking existence of /builds/some-project/schema.prisma +0ms
prisma:getSchema Reading schema from single file /builds/some-project/schema.prisma +1ms
prisma:getSchema Checking existence of /builds/some-project/prisma/schema.prisma +0ms
prisma:getSchema Reading schema from single file /builds/some-project/prisma/schema.prisma +1ms
prisma:getSchema Reading schema from multiple files /builds/some-project/prisma/schema +1ms
Prisma schema loaded from prisma/schema.prisma
prisma:getConfig Using getConfig Wasm +1ms
prisma:getConfig config data retrieved without errors in getConfig Wasm +47ms
prisma:get-platform Found distro info:
{
  "targetDistro": "musl",
  "familyDistro": "alpine",
  "originalDistro": "alpine"
} +2ms
prisma:get-platform Don't know any platform-specific paths for "alpine" on x64 (x86_64) +0ms
prisma:get-platform Falling back to "ldconfig" and other generic paths +1ms
prisma:get-platform Found libssl.so file using "ldconfig" or other generic paths: libssl.so.3 +4ms
prisma:get-platform The parsed libssl version is: 3.0.x +1ms
prisma:getSchema Reading schema from single file /builds/some-project/prisma/schema.prisma +1ms
prisma:fetch-engine:download The checksum file: /root/.cache/prisma/master/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/libquery-engine.sha256 is missing but this was ignored as the PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING environment variable is truthy. +2ms
prisma:getConfig Using getConfig Wasm +1ms
prisma:getConfig config data retrieved without errors in getConfig Wasm +18ms
prisma:getDMMF Using getDmmf Wasm +1ms
prisma:getDMMF Using given datamodel +1ms
prisma:getDMMF dmmf data retrieved without errors in getDmmf Wasm +415ms
prisma:generator prismaCLIDir /builds/some-project/node_modules/prisma +8ms
prisma:generator prismaClientDir /builds/some-project/node_modules/@prisma/client +0ms
prisma:generator baseDir /builds/some-project/prisma +0ms
prisma:generator typescriptPath /builds/some-project/node_modules/typescript +2ms
prisma:GeneratorProcess prisma:client:generator requiredEngine: libqueryEngine +34ms +125ms
prisma:mergeSchemas Using mergeSchemas Wasm +1ms
prisma:loadEnv project root found at /builds/some-project/node_modules/@prisma/client/package.json +34ms
prisma:getSchema prismaConfig {
  "packagePath": "/builds/some-project/package.json"
} +1ms
prisma:getGenerators neededVersions {
  "4c784e32044a8a016d99474bd02a3b6123742169": {
    "engines": [
      "libqueryEngine"
    ],
    "binaryTargets": [
      {
        "fromEnvVar": null,
        "value": "linux-musl-openssl-3.0.x",
        "native": true
      }
    ]
  }
} +0ms
prisma:fetch-engine:download The checksum file: /root/.cache/prisma/master/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/libquery-engine.sha256 is missing but this was ignored as the PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING environment variable is truthy. +2ms
prisma:getGenerators {
  "generatorBinaryPaths": {
    "libqueryEngine": {
      "linux-musl-openssl-3.0.x": "/builds/some-project/node_modules/prisma/libquery_engine-linux-musl-openssl-3.0.x.so.node"
    }
  }
} +0ms
prisma:tryLoadEnv Environment variables not found at null +11ms
prisma:tryLoadEnv Environment variables not found at undefined +0ms
prisma:tryLoadEnv No Environment variables loaded +0ms
Error: Error: 
ENOENT: no such file or directory, copyfile '/builds/some-project/node_modules/prisma/libquery_engine-linux-musl-openssl-3.0.x.so.node' -> '/builds/some-project/node_modules/.prisma/client/libquery_engine-linux-musl-openssl-3.0.x.so.node'
    at Sm.parse (/builds/some-project/node_modules/prisma/build/index.js:1492:71)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async lPt (/builds/some-project/node_modules/prisma/build/index.js:1999:1678)

The workaround with deleting cache no longer works with openssl3:

npm ci
rm -f node_modules/prisma/*.node
rm -rf ~/.cache/prisma
prisma generate

results in:

$ npm ci
added 816 packages in 3m

86 packages are looking for funding
  run `npm fund` for details
$ rm -f node_modules/prisma/*.node
$ rm -rf ~/.cache/prisma
$ npm run prisma:generate

> my-project@1.9.4 prisma:generate
> prisma generate

prisma:engines binaries to download libquery-engine, schema-engine +289ms
prisma:get-platform Found distro info:
{
  "targetDistro": "musl",
  "familyDistro": "alpine",
  "originalDistro": "alpine"
} +6ms
prisma:get-platform Don't know any platform-specific paths for "alpine" on x64 (x86_64) +0ms
prisma:get-platform Falling back to "ldconfig" and other generic paths +0ms
prisma:get-platform Found libssl.so file using "ldconfig" or other generic paths: libssl.so.3 +10ms
prisma:get-platform The parsed libssl version is: 3.0.x +0ms
prisma:fetch-engine:download file /builds/my-project/node_modules/@prisma/engines/libquery_engine-linux-musl-openssl-3.0.x.so.node exists but its version is libquery-engine  and we expect 4c784e32044a8a016d99474bd02a3b6123742169 +10ms
prisma:fetch-engine:download file /builds/my-project/node_modules/@prisma/engines/schema-engine-linux-musl-openssl-3.0.x exists but its version is schema-engine-cli  and we expect 4c784e32044a8a016d99474bd02a3b6123742169 +12ms
prisma:fetch-engine:download https://artifactory.example.com/artifactory/prisma-remote/all_commits/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/libquery_engine.so.node.gz will be downloaded to /builds/my-project/node_modules/@prisma/engines/libquery_engine-linux-musl-openssl-3.0.x.so.node +2ms
prisma:fetch-engine:download https://artifactory.example.com/artifactory/prisma-remote/all_commits/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/schema-engine.gz will be downloaded to /builds/my-project/node_modules/@prisma/engines/schema-engine-linux-musl-openssl-3.0.x +0ms
prisma:fetch-engine:download Downloading https://artifactory.example.com/artifactory/prisma-remote/all_commits/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/schema-engine.gz to /builds/my-project/node_modules/@prisma/engines/schema-engine-linux-musl-openssl-3.0.x ... +1ms
prisma:fetch-engine:download Downloading https://artifactory.example.com/artifactory/prisma-remote/all_commits/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/libquery_engine.so.node.gz to /builds/my-project/node_modules/@prisma/engines/libquery_engine-linux-musl-openssl-3.0.x.so.node ... +61ms
prisma:fetch-engine:downloadZip fetchChecksum() failed and was ignored as the PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING environment variable is truthy.
Error: Error: Failed to fetch sha256 checksum at https://artifactory.example.com/artifactory/prisma-remote/all_commits/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/libquery_engine.so.node.sha256 - 404  +60ms
prisma:fetch-engine:downloadZip fetchChecksum() failed and was ignored as the PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING environment variable is truthy.
Error: Error: Failed to fetch sha256 checksum at https://artifactory.example.com/artifactory/prisma-remote/all_commits/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/schema-engine.sha256 - 404  +4ms
prisma:loadEnv project root found at /builds/my-project/package.json +969ms
prisma:getSchema prismaConfig {
  "packagePath": "/builds/my-project/package.json"
} +1ms
prisma:tryLoadEnv Environment variables not found at null +1ms
prisma:tryLoadEnv Environment variables not found at undefined +0ms
prisma:tryLoadEnv No Environment variables loaded +0ms
prisma:getSchema prismaConfig {
  "packagePath": "/builds/my-project/package.json"
} +0ms
prisma:getSchema Checking existence of /builds/my-project/schema.prisma +1ms
prisma:getSchema Reading schema from single file /builds/my-project/schema.prisma +0ms
prisma:getSchema Checking existence of /builds/my-project/prisma/schema.prisma +0ms
prisma:getSchema Reading schema from single file /builds/my-project/prisma/schema.prisma +0ms
prisma:getSchema Reading schema from multiple files /builds/my-project/prisma/schema +2ms
Prisma schema loaded from prisma/schema.prisma
prisma:getConfig Using getConfig Wasm +0ms
prisma:getConfig config data retrieved without errors in getConfig Wasm +37ms
prisma:get-platform Found distro info:
{
  "targetDistro": "musl",
  "familyDistro": "alpine",
  "originalDistro": "alpine"
} +2ms
prisma:get-platform Don't know any platform-specific paths for "alpine" on x64 (x86_64) +1ms
prisma:get-platform Falling back to "ldconfig" and other generic paths +0ms
prisma:get-platform Found libssl.so file using "ldconfig" or other generic paths: libssl.so.3 +7ms
prisma:get-platform The parsed libssl version is: 3.0.x +0ms
prisma:getSchema Reading schema from single file /builds/my-project/prisma/schema.prisma +1ms
prisma:fetch-engine:download The checksum file: /root/.cache/prisma/master/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/libquery-engine.sha256 is missing but this was ignored as the PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING environment variable is truthy. +2ms
prisma:getConfig Using getConfig Wasm +1ms
prisma:getConfig config data retrieved without errors in getConfig Wasm +14ms
prisma:getDMMF Using getDmmf Wasm +1ms
prisma:getDMMF Using given datamodel +0ms
prisma:getDMMF dmmf data retrieved without errors in getDmmf Wasm +396ms
prisma:generator prismaCLIDir /builds/my-project/node_modules/prisma +6ms
prisma:generator prismaClientDir /builds/my-project/node_modules/@prisma/client +0ms
prisma:generator baseDir /builds/my-project/prisma +0ms
prisma:generator typescriptPath /builds/my-project/node_modules/typescript +1ms
prisma:GeneratorProcess prisma:client:generator requiredEngine: libqueryEngine +31ms +104ms
prisma:mergeSchemas Using mergeSchemas Wasm +1ms
prisma:loadEnv project root found at /builds/my-project/node_modules/@prisma/client/package.json +32ms
prisma:getSchema prismaConfig {
  "packagePath": "/builds/my-project/package.json"
} +1ms
prisma:getGenerators neededVersions {
  "4c784e32044a8a016d99474bd02a3b6123742169": {
    "engines": [
      "libqueryEngine"
    ],
    "binaryTargets": [
      {
        "fromEnvVar": null,
        "value": "linux-musl-openssl-3.0.x",
        "native": true
      }
    ]
  }
} +0ms
prisma:fetch-engine:download The checksum file: /root/.cache/prisma/master/4c784e32044a8a016d99474bd02a3b6123742169/linux-musl-openssl-3.0.x/libquery-engine.sha256 is missing but this was ignored as the PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING environment variable is truthy. +1ms
prisma:getGenerators {
  "generatorBinaryPaths": {
    "libqueryEngine": {
      "linux-musl-openssl-3.0.x": "/builds/my-project/node_modules/prisma/libquery_engine-linux-musl-openssl-3.0.x.so.node"
    }
  }
} +0ms
prisma:tryLoadEnv Environment variables not found at null +10ms
prisma:tryLoadEnv Environment variables not found at undefined +0ms
prisma:tryLoadEnv No Environment variables loaded +1ms
Error: Error: 
ENOENT: no such file or directory, copyfile '/builds/my-project/node_modules/prisma/libquery_engine-linux-musl-openssl-3.0.x.so.node' -> '/builds/my-project/node_modules/.prisma/client/libquery_engine-linux-musl-openssl-3.0.x.so.node'
    at Sm.parse (/builds/my-project/node_modules/prisma/build/index.js:1492:71)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async lPt (/builds/my-project/node_modules/prisma/build/index.js:1999:1678)