[Bug?]: Checksum mismatch for github package - using yarn v1 to pack

lexanth commented 2 years ago

Self-service

[ ] I'd be willing to implement a fix

Describe the bug

When installing a package from github "cypress-log-to-output": "Narretz/cypress-log-to-output#feat-no-output", I'm getting a The remote archive doesn't match the expected checksum error in CI (CircleCI - docker image cimg/node:16.13.2). I'm using the node modules linker, not PnP.

I've compared the contents of the package it's creating in /tmp on CI with the contents of the package in my local (macOS) yarn cache. The only file with checksum differences is package.json. In my local version, the package.json has the trailing newline (which matches the git repo). The version on CircleCI is trimming the trailing newline as part of the yarn pack --install-if-required step (the git clone still has the newline before and after the yarn pack, but the version inside the tarball seems to have it trimmed).

From looking at the source, I can see that yarn berry will always be trimming this trailing newline and I can reproduce it locally.

If I switch from the github protocol to the git one, I get the same result, but can see the package being regenerated:

➤ YN0000: ┌ Resolution step
➤ YN0013: │ cypress-log-to-output@git@github.com:Narretz/cypress-log-to-output.git#commit=58931ea7110dfd7779814f699aa12006eea93585 can't be found in the cache and will be fetched from the remote repository
➤ YN0000: │ /private/var/folders/r5/lxs794h569vg8ntxmrn8jrvm0000gp/T/xfs-fc04a1e2 STDOUT Packing cypress-log-to-output@git@github.com:Narretz/cypress-log-to-output.git#commit=58931ea7110dfd7779814f699aa12006eea93585 from sources
➤ YN0000: │ /private/var/folders/r5/lxs794h569vg8ntxmrn8jrvm0000gp/T/xfs-fc04a1e2 STDOUT No package manager configuration detected; defaulting to Yarn
➤ YN0000: │ /private/var/folders/r5/lxs794h569vg8ntxmrn8jrvm0000gp/T/xfs-fc04a1e2 STDOUT
➤ YN0000: │ /private/var/folders/r5/lxs794h569vg8ntxmrn8jrvm0000gp/T/xfs-fc04a1e2 STDOUT yarn pack v1.22.15
➤ YN0000: │ /private/var/folders/r5/lxs794h569vg8ntxmrn8jrvm0000gp/T/xfs-fc04a1e2 STDOUT success Wrote tarball to "/private/var/folders/r5/lxs794h569vg8ntxmrn8jrvm0000gp/T/xfs-fc04a1e2/package.tgz".
➤ YN0000: │ /private/var/folders/r5/lxs794h569vg8ntxmrn8jrvm0000gp/T/xfs-fc04a1e2 STDOUT Done in 0.06s.

Note specifically yarn pack v1.22.15

I'm definitely running yarn berry, but for the yarn pack inside it, it's using the globally installed yarn 1. I'm guessing yarn 1 isn't trimming the trailing newline, so then we get a mismatch.

In the source, in yarn 2 mode it seems to be doing a lot to try to get this environment set up right, but for some reason (nvm?) it isn't working for me locally. It does seem to the right thing in my CI environment. I used corepack enable to set up berry - maybe that isn't ending up being carried across when spawning the separate yarn process when using nvm?

This seems like some missing piece of local environment config, but I've followed the getting started docs and can't see anything else missing.

To reproduce

Can't reproduce with sherlock - depends on the yarn 1 install

Create new package with yarn init -2

Add a github dependency which doesn't define a package manager

{
"name": "repro",
"packageManager": "yarn@3.1.1",
"dependencies": {
"cypress-log-to-output": "Narretz/cypress-log-to-output#feat-no-output"
}
}

Set enableInlineBuilds: true so that you can see the pack output
Run yarn install
Note that it uses yarn 1.x to do yarn pack on the github dependency

Environment

System:
    OS: macOS 10.15.7
    CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
  Binaries:
    Node: 16.13.0 - /private/var/folders/r5/lxs794h569vg8ntxmrn8jrvm0000gp/T/xfs-a816c7ae/node
    Yarn: 3.1.1 - /private/var/folders/r5/lxs794h569vg8ntxmrn8jrvm0000gp/T/xfs-a816c7ae/yarn
    npm: 8.1.0 - ~/.nvm/versions/node/v16.13.0/bin/npm
  npmPackages:
    jest: ^27.2.5 => 27.4.5

Additional context

No response

yarnbot commented 2 years ago

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟

dwickern commented 2 years ago

I had the same issue running yarn@3.2.2 (current stable version) installed with corepack.

Installing the package "canvas": "dwickern/canvas-noop" from yarn 1.22.19: canvas-https-d4dab75653-31cdc4ba4d.zip

The same package installed with yarn 3.2.2 via corepack without the yarn v1 bootstrapper: canvas-https-d4dab75653-6b6d48e4ad.zip

Workaround is to corepack disable and install yarn v1.

TuurDutoit commented 2 years ago

I have encountered the same problem while upgrading from Yarn v1 to v3.2.1 recently.

In package.json (only relevant parts):

{
  "packageManager": "yarn@3.2.1",
  "dependencies": {
    "timecop": "git+https://github.com/jamesarosen/Timecop.js.git#d421df9b00386266bda93eb9c2ea15fedc888761"
  }
}

When installing on my machine (Mac OS) with corepack enabled:

$ rm -rf node_modules && yarn cache clean --all && yarn | grep '/private/var/folders/fk'
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-7e3c703f STDOUT Packing timecop@https://github.com/jamesarosen/Timecop.js.git#commit=d421df9b00386266bda93eb9c2ea15fedc888761 from sources
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-7e3c703f STDOUT No package manager configuration detected; defaulting to Yarn
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-7e3c703f STDOUT 
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-7e3c703f STDOUT yarn pack v1.22.15
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-7e3c703f STDOUT success Wrote tarball to "/private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-7e3c703f/package.tgz".
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-7e3c703f STDOUT Done in 0.18s.

With corepack disabled:

$ rm -rf node_modules && yarn cache clean --all && yarn | grep '/private/var/folders/fk'
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT Packing timecop@https://github.com/jamesarosen/Timecop.js.git#commit=d421df9b00386266bda93eb9c2ea15fedc888761 from sources
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT No package manager configuration detected; defaulting to Yarn
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT 
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: .jshintrc
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: Contributing.md
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: Gemfile
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: Gemfile.lock
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: LICENSE
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: README.md
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: Rakefile
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: lib/BuildTemplate.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: lib/MockDate.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: lib/TimeStackItem.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: lib/Timecop.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: lib/timecop/compilation_task.rb
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: lib/timecop/jshint_task.rb
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: package.json
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/MockDateSpec.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/TimeStackItemSpec.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/TimecopSpec.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/helpers/SpecHelper.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/support/jasmine-html.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/support/jasmine.css
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/support/jasmine.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/support/jasmine.yml
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/support/jquery-1.4.4.min.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: spec/javascripts/support/json2.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: timecop.js
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: Package archive generated in /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064/package.tgz
➤ YN0000: │ /private/var/folders/fk/c9z1tyhj02nfdn9dz_71vjk80000gq/T/xfs-88fd0064 STDOUT ➤ YN0000: Done in 0s 112ms

The resulting ZIP files are almost exactly identical, save for one difference: an extra newline in the package.json. Note that, in both cases, the version of Node and Yarn CLI is exactly the same. The only difference is that when corepack is enabled/disabled, different versions of Yarn are available (I assume), which means our Git dependency is built with a slightly different version of Yarn, which means the newline is removed / added, which means the checksum changes, which means the install fails. In summary: depending on the Yarn versions available in your environment, the checksums of Git deps may be different.

I ran into this in practice in 2 instances:

I installed a package on my machine (corepack enabled). Install fails on the CI (node:16.15.1-alpine, corepack disabled by default)
I installed a package on my machine (corepack enabled). A colleague pulls my changes to their machine (corepack disabled)

For the time being, we'll avoid using corepack at all so all environments are the same.

yarnpkg / berry