`EROFS: read-only file system, chmod '/var/task/generated/prisma-client/runtime/query-engine-rhel-openssl-1.0.x'`

[janpio]

I followed the setup in https://github.com/prisma/prisma2-e2e-tests/tree/master/platforms/netlify (clone, remove .netlify/state.json, yarn install, yarn prisma2 generate, execute commands from build.sh manually (zip does not exist and has to be done manually), get deployed site and function!).

When then calling the function, I get:

{"errorType":"Error","errorMessage":"\nInvalid `prisma.()` invocation in\n/var/task/generated/prisma-client/index.js:323:37\n\n\n\nEROFS: read-only file system, chmod '/var/task/generated/prisma-client/runtime/query-engine-rhel-openssl-1.0.x'","trace":["Error: ","Invalid `prisma.()` invocation in","/var/task/generated/prisma-client/index.js:323:37","","","","EROFS: read-only file system, chmod '/var/task/generated/prisma-client/runtime/query-engine-rhel-openssl-1.0.x'","    at PrismaClientFetcher.request (/var/task/generated/prisma-client/index.js:60:27)"]}

https://goofy-tereshkova-4bf91a.netlify.com/.netlify/functions/index

Theory: Because the the generated client was created on Windows, it is missing the necessary file permission flags.

(index.zip that was uploaded: index.zip)

[simonobe]

this is duplicate of prisma/prisma-client-js#445 ... btw: I am also building on windows

[janpio]

Thanks for confirming @simonobe, so we indeed have a Windows problem here. Another description of the problem on AWS Lambda was here: https://github.com/prisma/prisma-client-js/issues/445

[simonobe]

Is there any estimate? Can I contribute in any way? Do you at least have any idea on how to solve this?

[divyenduz]

In my understanding, this issue happens when the build machine is an OS without binary flags (like windows) and the deployment machine is a machine with binary flags.

If you specify a binary provider in your Prisma schema file to download the binary for target platform OS, like in the case of netlify:

binaryTargets = ["native", "rhel-openssl-1.0.x"]

The binary download happens with prisma2 generate on the build OS (windows). But the downloaded binary doesn't have any executable flag as it is on windows.

When this binary is deployed a Prisma client attempts to run it, it fails. Most likely on this line. This would usually just work but on lambda it doesn't work because this part of the filesystem is not writeable.

---

Next steps: We need to validate the above hypothesis and solve this in some way (already have +x flag on binaries in S3 or downloading the files in /tmp (writeable for lambda) adding the +x flag and copying the binary or something else)

[divyenduz]

We need to validate the above hypothesis

I can confirm it. I tried deploying to Lambda from Windows are ran into this error.

Now, we need to think of a way to fix this. Maybe, detecting lambda runtime, copying the binary to /tmp, adding the required flag and copying it back (or a variation of this is needed)

[kyunghoon]

i'm using lambda-layers to package the runtime which copies it into/opt/ is there a configuration setting in prisma to specify the location of the binary?

[422158]

We need to validate the above hypothesis

I can confirm it. I tried deploying to Lambda from Windows are ran into this error.

Now, we need to think of a way to fix this. Maybe, detecting lambda runtime, copying the binary to /tmp, adding the required flag and copying it back (or a variation of this is needed)

how do you intend to copy it back when it is read only fs

i'm using lambda-layers to package the runtime which copies it into/opt/ is there a configuration setting in prisma to specify the location of the binary?

yeah i think this guy might be onto something. i am not sure but i think that within lambda environemnt you can only change fs within /tmp... if we would be able to change/specify the binary location within prisma client, we might copy the binary to the /tmp, add execution rights and use this modified executable in client

[divyenduz]

or a variation of this is needed

Yup, we can use the binary from /tmp, if needed.

[divyenduz]

You can also already configure custom binary paths and work around this: https://github.com/prisma/specs/tree/master/binaries#environment-variables

[janpio]

i'm using lambda-layers to package the runtime which copies it into/opt/ is there a configuration setting in prisma to specify the location of the binary?

yeah i think this guy might be onto something. i am not sure but i think that within lambda environemnt you can only change fs within /tmp... if we would be able to change/specify the binary location within prisma client, we might copy the binary to the /tmp, add execution rights and use this modified executable in client

This is what Prisma Client could/should do itself possibly if it runs into the error message I specified in the issue title. Right now it fails if it can not set +x on the file. But trying to copy to /tmp might give us another way to fix this and thus also fix this issue in general.

We will look into this.

[divyenduz]

Listing down the potential workarounds for this:

1. Use a CI/CD setup, Building your application in a *nix based OS with executable flags will work.

2. Use custom binary paths environment variables and upload the binary to /tmp and alter its executable flag. When we fix this, this is probably we will do internally.

3. Use lambda layers and make the binaries be available at a path. Use the path at runtime. Optionally, delete the uploading binaries to make the build smaller.

[kyunghoon]

following @divyenduz tip, i have set the environment variable PRISMA_QUERY_ENGINE_BINARY to point to the binary, in my case /opt/runtime/query-engine-rhel-openssl-1.0.x. but now i get a different and less useful error.

please let me know if there is anything i can do to help.

{"errorType":"Error","errorMessage":"\nInvalidprisma.users.create()invocation in\n/var/task/index.js:5506:25\n\nNo such file or directory (os error 2)","stack":["Error: ","Invalidprisma.users.create()invocation in","/var/task/index.js:5506:25","","No such file or directory (os error 2)"," at PrismaClientFetcher.request (/var/task/index.js:4122:17)"," at processTicksAndRejections (internal/process/task_queues.js:94:5)"]}

[janpio]

Please create a new issue for this @kyunghoon - we will look into it. Thanks.

[leungandrew]

I've experienced this issue when deploying a Nexus + Prisma app to serverless using the Serverless Express component. Reading through the comments, it seems people have isolated it to be a Windows issue? But I am also seeing this issue when compiling via darwin, and deploying with

generator prisma_client {
  provider      = "prisma-client-js"
  binaryTargets = ["native", "rhel-openssl-1.0.x"]
}

[eddeee888]

I have got Prisma client working with Serverless Express component. Here's how:

1. Deploy a lambda layer containing only your Prisma client for. Here's a sample Github Action on how to do it

2. Use the uploaded lambda layer ARN in your Express serverless.yml

3. Deploy Express lambda

4. Check the Express Lambda function on AWS. If you can see that it uses the Prisma layer, it means it works!

( OPTIONAL ) It's better IMO to only install binary target based on the environment so the lambda layer is smaller. I normally use env to install different binary, depending on the environment ( native for devs, rhel-openssl-1.0.x when building for prod )

[divyenduz]

I am able to reproduce this thanks to the reproduction provided by @leungandrew (https://github.com/leungandrew/nexus-prisma-serverless-express)

It was previously established that this is a windows -> Linux issue but I am able to reproduce this between mac -> Linux. I also checked that the local binary before deployment has the +x flag

This issue looks different from the contents of this issue, where it is clear that the executable flag is lost on windows, hence, we are tracking it separately here: https://github.com/prisma/prisma-client-js/issues/751

[abelchee]

Copy to tmp folder works for me.

const binaryPath = '/tmp/query-engine-rhel-openssl-1.0.x';
if (!fs.existsSync(binaryPath)) {
  const { spawnSync } = require('child_process');

  spawnSync('cp', [
    `${process.env.LAMBDA_TASK_ROOT}/node_modules/.prisma/client/query-engine-rhel-openssl-1.0.x`,
    '/tmp/',
  ]);

  spawnSync('chmod', [`555`, '/tmp/query-engine-rhel-openssl-1.0.x']);
}

And need an env:

  PRISMA_QUERY_ENGINE_BINARY: /tmp/query-engine-rhel-openssl-1.0.x

[samrith-s]

Copy to tmp folder works for me.

const binaryPath = '/tmp/query-engine-rhel-openssl-1.0.x';
if (!fs.existsSync(binaryPath)) {
  const { spawnSync } = require('child_process');

  spawnSync('cp', [
    `${process.env.LAMBDA_TASK_ROOT}/node_modules/.prisma/client/query-engine-rhel-openssl-1.0.x`,
    '/tmp/',
  ]);

  spawnSync('chmod', [`555`, '/tmp/query-engine-rhel-openssl-1.0.x']);
}

And need an env:

  PRISMA_QUERY_ENGINE_BINARY: /tmp/query-engine-rhel-openssl-1.0.x

How and where are you running this? I am trying to deploy my nest application and got into the same issue.

[davidboom95]

Copy to tmp folder works for me.

const binaryPath = '/tmp/query-engine-rhel-openssl-1.0.x';
if (!fs.existsSync(binaryPath)) {
  const { spawnSync } = require('child_process');

  spawnSync('cp', [
    `${process.env.LAMBDA_TASK_ROOT}/node_modules/.prisma/client/query-engine-rhel-openssl-1.0.x`,
    '/tmp/',
  ]);

  spawnSync('chmod', [`555`, '/tmp/query-engine-rhel-openssl-1.0.x']);
}

And need an env:

  PRISMA_QUERY_ENGINE_BINARY: /tmp/query-engine-rhel-openssl-1.0.x

How and where are you running this? I am trying to deploy my nest application and got into the same issue.

+1

[samrith-s]

I've tried running this snippet in my lambda handler and have set the environment variable as well. But to no avail.

I don't get any logs of this running, and I get an error saying it cannot find it in /tmp. Stuck very badly with this.

[davidboom95]

I've tried running this snippet in my lambda handler and have set the environment variable as well. But to no avail.

I don't get any logs of this running, and I get an error saying it cannot find it in /tmp. Stuck very badly with this.

I'm using a serverless monorepo, so I'm having just one root prisma file and multiple serverless.yml services. Moving that 'schema.prisma' and 'query-engine-rhel-openssl-1.0.x' between folders is a nightmare

[samrith-s]

Are you using NestJS? Or webpack? If yes, I can share what I have done.

[davidboom95]

I solved it. I made a file with this code. This code gets executed on every lambda request. To get the file paths, I needed to debug and print folders and its contents to know where the files are.

const fs = require('fs');

const binaryPath = '/tmp/query-engine-rhel-openssl-1.0.x'

if (!fs.existsSync(binaryPath) && !process.env.IS_OFFLINE) {

  fs.copyFileSync(
    `${process.env.LAMBDA_TASK_ROOT}/query-engine-rhel-openssl-1.0.x`,
    binaryPath
  )

  fs.chmodSync(binaryPath, 555)

  process.env.PRISMA_QUERY_ENGINE_BINARY = binaryPath
}

[samrith-s]

@davidboom95 i would suggest using Lambda Layers (which is what I'm using) to greatly reduce bundle size and avoid doing this which also results in better startup times :)

[davidboom95]

Do you have the code? I tried to use a lambda layer but didn't know how to execute that preparation automatically

[samrith-s]

Yes. Can you message me on the Prisma Slack channel?

[larrybek]

@divyenduz @samrith-s. Thanks for the suggestion it worked great, if you had got deeper with layer usage will be happy if you can share something on it. Will be glad.

[samrith-s]

@larrybek sure! Message me on Slack and I can share my Serverless file :)

[MichalBunkowski]

@larrybek sure! Message me on Slack and I can share my Serverless file :)

Hi @samrith-s ! Could you share your serverless file please?

[janpio]

I confirmed the original problem (build on Windows, deploy to Lambda, fails) does still exist. Lambda logs:

START RequestId: 0827675d-b212-4165-89cd-17dbb67e1626 Version: $LATEST
2021-03-24T22:55:41.640Z    0827675d-b212-4165-89cd-17dbb67e1626    ERROR   Invoke Error    ***"errorType":"Error","errorMessage":"\nInvalid `prisma.user.deleteMany()` invocation:\n\n\n  EROFS: read-only file system, chmod '/var/task/node_modules/.prisma/client/query-engine-rhel-openssl-1.0.x'","code":"EROFS","clientVersion":"2.20.0-dev.20","stack":["Error: ","Invalid `prisma.user.deleteMany()` invocation:","","","  EROFS: read-only file system, chmod '/var/task/node_modules/.prisma/client/query-engine-rhel-openssl-1.0.x'","    at cb (/var/task/node_modules/@prisma/client/runtime/index.js:34788:17)"]***
END RequestId: 0827675d-b212-4165-89cd-17dbb67e1626
REPORT RequestId: 0827675d-b212-4165-89cd-17dbb67e1626  Duration: 360.04 ms Billed Duration: 361 ms Memory Size: 128 MB Max Memory Used: 84 MB  Init Duration: 354.88 ms    

(All comments regarding Serverless and related really belong into https://github.com/prisma/prisma/issues/5053 which is not about the Windows problem)

[janpio]

Interesting that https://github.com/prisma/e2e-tests/pull/1539 shows that Serverless Framework, deployed to Lambda from Windows does not show the same failure - so they are doing something to fix that problem.

[Jolg42]

@janpio indeed, they seem to chmod to 755 for windows here when zipping https://github.com/serverless/serverless/blob/6e3e21cafaa053a0f30239513eac5c7a725b7694/lib/plugins/package/lib/zipService.js#L92-L99

[janpio]

TIL you can set the permission of a file when putting it into a zip file 👍

[pantharshit00]

@janpio Is this still actionable?

[janpio]

Yes, as far as I know this problem still exists when built on Windows as the file does not got the executable flag when uploaded to the platform where this matters.