Binary right is not preserved with pnpm

gregberge commented 5 months ago

Describe the bug

Using pnpm the binary right is not preserved. For example, the odiff package is broken.

By investigating I found that the underlying binary used in the package has lost its execution right after deploying on Heroku:

Fatal error: exception Unix.Unix_error(Unix.EACCES, "execve", "/app/node_modules/.pnpm/odiff-bin@2.6.1/node_modules/odiff-bin/3/i/odiff-33ddbef3/bin/ODiffBin")

To Reproduce

Install odiff
Run a script on Heroku that use it

Versions (please complete the following information):

Heroku Stack: Heroku-22
Node Version: 20
NPM or Yarn Version: pnpm v9
Buildpack Version: v245
Additional context Add any other context about the problem here.

colincasey commented 5 months ago

Hi @gregberge, I'm having trouble reproducing this bug to investigate further. Can you provide a bit more information around how you're calling odiff?

As a quick test, I tried deploying an app using the following package.json file + some image files:

{
  "name": "issue-1247",
  "version": "0.0.0",
  "dependencies": {
    "odiff-bin": "2.6.1"
  },
  "engines": {
    "node": "20.x"
  },
  "scripts": {
    "build": "odiff tiger.jpg tiger-2.jpg output.jpg"
  },
  "packageManager": "pnpm@9.0.6+sha256.0624e30eff866cdeb363b15061bdb7fd9425b17bc1bb42c22f5f4efdea21f6b3"
}

The command works though the build step fails because odiff reports that the images are different:

remote: -----> Build
remote:        Running build
remote:
remote:        > pnpm-engine@1.0.0 build /tmp/build_6711e685
remote:        > odiff tiger.jpg tiger-2.jpg output.jpg
remote:
remote:        Failure! Images are different.
remote:        Different pixels: 7586 (1.137331%)
remote:         ELIFECYCLE  Command failed with exit code 22.

haubey commented 5 months ago

We are also running into this issue with sentry/cli. It seems to work when the package is included in dependencies but doesn't work (EACCES) when added in devDependencies.

colincasey commented 5 months ago

@haubey Can you provide details on how you're calling @sentry/cli?

haubey commented 5 months ago

@colincasey sure thing.

Before pnpm was installed via corepack we had the following build script (pared down)

NODE_ENV=production
npm i -g pnpm@8.15.3

pnpm i --prod --frozen-lockfile

set +e # Prevents a Sentry error from stalling the Heroku deployment

# Ensures that sourcemaps are only uploaded from Heroku
if [ -n "${SOURCE_VERSION}" ]; then
  pnpm i -g @sentry/cli
  sentry-cli releases new $SOURCE_VERSION
  sentry-cli releases  files $SOURCE_VERSION upload-sourcemaps ./packages/our-package/build
  sentry-cli releases finalize $SOURCE_VERSION
fi

# Prevents a sentry-cli failure from crashing the Heroku builds
exit 0

And it worked fine. With the codepack change we removed npm i -g pnpm@8.15.3 and added it as an engine in our package.json instead. At that point the pnpm i -g @sentry/cli call started not being runnable. We then moved it to dependencies and it worked, then devDependencies and it failed. However we're also experiencing some issues around our pnpm cache with the move to engines/packageManager so I can't say for certain what the exact issue we're experiencing is. Once we nail it down, I will be sure to follow up here!

gregberge commented 3 months ago

Gently bumping this one. @colincasey any news on that?

colincasey commented 3 months ago

@gregberge the only update here is that I also tried to reproduce the issue using the information provided by @haubey. The results from that are summarized below:

Package	Declared In	Command	Build time	Run time
`@sentry/cli`	`dependencies`	`sentry-cli -V`	Success	Success
`@sentry/cli`	`devDependencies`	`sentry-cli -V`	Success	Error - Not found

[!NOTE] The failure at run time when it's installed as a dev dependency is expected since, by default, dev dependencies are pruned at the end of the build.

At no point did I observe any EACCES issues calling the @sentry/cli binary installed by pnpm.

To investigate this further, it would help if you could provide some information around how you're calling odiff or share a minimal reproduction of the issue.

dzuelke commented 3 weeks ago

Do y'all by any chance have node_modules checked into Git, @gregberge and @haubey ?

dzuelke commented 3 weeks ago

Actually, could it be https://github.com/pnpm/pnpm/issues/6285, or more specifically this: https://github.com/aws/aws-pdk/issues/322#issuecomment-1484245293?

It appears PNPM does this intentionally for portability reasons and requires executables to either be listed in bin or publishConfig.executableFiles in package.json.

haubey commented 3 weeks ago

@dzuelke we do not have node_modules checked into git. For now we've turned caching off and things are generally working for us, though some services like Fontawesome charge for bandwidth so it would still be good to figure out a more cache-able solution.

colincasey commented 3 weeks ago

I've been able to make a bit of headway here in terms of a minimal reproduction on the Heroku platform. I'm going to record my notes here.

Reproducing

Create a new folder containing the following package.json:

{
  "name": "eaccess-test",
  "private": true,
  "scripts": {
    "build": "sentry-cli --version"
  },
  "packageManager": "pnpm@8.6.1+sha512.3f9e23f20bdbf7a27b87b804383f1dafdb5cb35cdc40fce590aff2215255446ff595878ee4f33429e6a0e7c3882b1ae926514f6fea6a5ba75e52f87bfc2592e7",
  "dependencies": {
    "@sentry/cli": "1.77.3"
  },
  "engines": {
    "node": "18.20.2"
  }
}

Setup the remaining files (e.g.; pnpm-lock.yaml, .gitignore):
```
pnpm install
echo "node_modules" > .gitignore
```

Create the Heroku app and deploy it:

heroku create
git add .
git commit -m "test eaccess app"
git push

Redeploy the app:

git commit --allow-empty -m "trigger build" 
git push

Observations

Builds using a fresh cache will succeed
Builds using a restored cache will encounter the EACCESS error
The @sentry/cli module uses a postinstall script to fetch an appropriate binary for the target architecture (the odiff-bin package mentioned in this issue also does something similar)
The permissions of the downloaded sentry-cli binary are as follows:

Cache State Permissions in
node_modules/.pnpm Permissions in
pnpm store

Fresh -rwx------ -rw-------

Restored -rw------- -rw-------

Cache State	Permissions in `node_modules/.pnpm`	Permissions in pnpm store
Fresh	`-rwx------`	`-rw-------`
Restored	`-rw-------`	`-rw-------`

I haven't been able to reproduce the above in a local environment (yet). Only on the Heroku platform. This situation has made isolating a root cause for this behavior more difficult and time consuming.

Workarounds

Either of the following workarounds will prevent the EACCESS error by forcing the postinstall hook to execute.

Disable the pnpm side-effects cache by adding an .npmrc to the application containing:
```
side-effects-cache=false
```
https://pnpm.io/npmrc#side-effects-cache
Disabling the Node.js modules cache:
```
heroku config:set NODE_MODULES_CACHE=false
```
https://devcenter.heroku.com/articles/nodejs-support#cache-behavior

heroku / heroku-buildpack-nodejs