[Bug?]: yarn patch generated unparsable patch: URL

lachlanhunt commented 2 years ago

Self-service

[ ] I'd be willing to implement a fix

Describe the bug

My company uses an internal package registry where all npm packages are fetched from. This is configured in ~/.yarnrc.yaml (company name replaced with example)

npmRegistries:
  "https://packages.example.com/artifactory/api/npm/npm-remote":
    npmAlwaysAuth: false
npmScopes:
  example:
    npmRegistryServer: "https://packages.example.com/artifactory/api/npm/example-npm"
    npmPublishRegistry: "https://packages.example.com/artifactory/api/npm/example-npm"

The result of this is that the resolution for the package in yarn.lock looks like this:

  resolution: "@example/eslint-plugin-example@npm:4.0.0::__archiveUrl=https%3A%2F%2Fpackages.example.com%2Fapi%2Fnpm%2Fexample-npm%2F%40example%2Feslint-plugin-example%2F-%2F%40example%2Feslint-plugin-example-4.0.0.tgz"

When I tried to patch this packing using:

$ yarn patch @example/eslint-plugin-example
# make edits
$ yarn patch-commit -s /private/var/folders/path/to/code/user
...

This generated a patch URL in package.json that looked like this:

```json
{
  ...
  "resolutions": {
    "@example/eslint-plugin-example@4.0.0": "patch:@example/eslint-plugin-example@npm:4.0.0::__archiveUrl=https%3A%2F%2Fpackages.example.com%2Fapi%2Fnpm%2Fexample-npm%2F%40example%2Feslint-plugin-example%2F-%2F%40example%2Feslint-plugin-example-4.0.0.tgz#.yarn/patches/@example-eslint-plugin-example-npm-4.0.0-bab1aa1587.patch"
  }

}

(Then I fixed the version range in that resolution as described in https://github.com/yarnpkg/berry/issues/4231#issuecomment-1072847435 )

Finally, running yarn to apply the patch gives this error:

$ yarn
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: Patch locators must explicitly define their source
    at xue (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:733:658)
    at dE (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:733:978)
    at iO.bindDescriptor (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:736:2747)
    at wd.bindDescriptor (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:391:1082)
    at wd.bindDescriptor (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:391:1082)
    at reduceDependency (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:390:29055)
    at tA.reduceHook (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:392:3584)
    at ee (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:439:6759)
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 23ms

I was able to work around this issue by modifying the patch: URL like this:

{
  ...
  "resolutions": {
    "@example/eslint-plugin-example@^4.0.0": "patch:@example/eslint-plugin-example@npm:4.0.0#.yarn/patches/@example-eslint-plugin-example-npm-4.0.0-bab1aa1587.patch"
  }
}

It appears the presence of ::__archiveUrl=https... breaks the parsing of the patch URL in yarn.

To reproduce

(as explained above)

edit: see reproduction in comment below https://github.com/yarnpkg/berry/issues/4711#issuecomment-1960736169

Environment

System:
  OS: macOS 12.5
  CPU: (10) arm64 Apple M1 Pro
Binaries:
  Node: 16.16.0 - /private/var/folders/78/vv5cyf6570xcqbsy0r2nfyfw0000gn/T/xfs-99d9804c/node
  Yarn: 3.2.2 - /private/var/folders/78/vv5cyf6570xcqbsy0r2nfyfw0000gn/T/xfs-99d9804c/yarn
  npm: 8.11.0 - ~/Library/Caches/fnm_multishells/21303_1659328957074/bin/npm

Additional context

No response

xenoterracide commented 2 years ago

hmm... I wonder if this is the same as the problem I just came here to report

xenoterracide commented 2 years ago

I don't get an error though, mine just silently doesn't apply

xenoterracide commented 2 years ago

linked my bug, seems very related to both that you mentioned, but mine seems to be both of the problems at once...

wojtekmaj commented 2 years ago

I was able to work around this issue by modifying the patch: URL like this:

OMG, I want to kiss you right now. I struggled with this for MONTHS, manually applying the patches.

yarnbot commented 2 years ago

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟

lachlanhunt commented 2 years ago

I couldn't figure out how exactly to reproduce this issue using Sherlock. Last time I tried to use it, the page failed to load the editor properly, so I gave up and assumed the tool was broken.

I explained the steps to reproduce in the description. I'll need a maintainer to help out.

wojtekmaj commented 2 years ago

This is my completely blind attempt at repro, as playground is not working, and local Sherlock does not expose (undocumented) methods like yarn or packageJsonAndInstall:

const installPromise = packageJsonAndInstall({
  dependencies: {
    [`@codertocat/hello-world-npm`]: `latest`,
  },
});

await expect(installPromise).resolves.toBeTruthy();

// Use node_modules resolver
// await yarn(`config`, `set nodeLinker node-modules`);

// Set npmRegistryServer
await yarn(`config`, `set npmRegistryServer "https://npm.pkg.github.com"`);

// Extract the package for patching
const patchOutput = await yarn(`patch`, `@codertocat/hello-world-npm`);
const patchOutputLines = patchOutput.split("\n");

// Look for "You can now edit the following folder: …"
const regex1 = /You can now edit the following folder: (.*)/;
const line1 = patchOutputLines.find((line) => line.match(regex1));
const folder = line1.match(regex1)[1];

// Make the edit in extracted package README.md file
const filePath = path.join(path.resolve(folder), `index.js`);
const file = await fs.readFile(filePath, `utf8`);
await fs.writeFile(
  filePath,
  file.replace("'Hello world!'", "'Hello world from patch!'")
);

// Look for "Once you are done run … and Yarn will store a patchfile based on your changes."
const regex2 =
  /Once you are done run (.*) and Yarn will store a patchfile based on your changes./;
const line2 = patchOutputLines.find((line) => line.match(regex2));
const command = line2.match(regex2)[1];
const [yarnText, commandText, ...args] = command.split(" ");

// Run the command
await yarn(commandText, ...args);

// Install repo again to apply patches
const install2Promise = await yarn(`install`);

/**
 * BUG #1:
 * Error: @codertocat/hello-world-npm@npm:1.0.2::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Codertocat%2Fhello-world-npm%2F1.0.2%2F9384a68bd287b859f028683bcaacedd27803f1dd isn't supported by any available resolver
 **/
expect(install2Promise).resolves.toBeTruthy();

// See if the patch was applied
const installedModule = require("@codertocat/hello-world-npm");

const output = installedModule.helloWorld();

/**
 * BUG #2: Patch not applied?
 */
expect(output).toEqual("Hello world from patch!");

arcanis commented 2 years ago

Mind trying the 4.0.0-rc (yarn set version canary)? Patches have been significantly improved there, and I'm unlikely to backport those changes to 3.x as some of them are backward-incompatible, and others just difficult to safely extract.

wojtekmaj commented 2 years ago

@arcanis I did the tests on 2 dimensions:

Yarn version: latest (3.2.3), canary (4.0.0-rc.18)
Dependency version range: exact ("@codertocat/hello-world-npm": "1.0.2") and caret ("@codertocat/hello-world-npm": "^1.0.0")

Here are the commands I ran:

# 4.0.0 ONLY:
yarn set version canary

yarn config set nodeLinker node-modules
yarn config set npmRegistryServer "https://npm.pkg.github.com"

# CARET ONLY:
yarn add @codertocat/hello-world-npm@^1.0.0
# EXACT ONLY:
yarn add @codertocat/hello-world-npm@1.0.2

yarn patch @codertocat/hello-world-npm
# (editing the file now by changing "Hello world!" to "Hello world from patch!" & saving it)
yarn patch-commit -s # (dynamically created folder path here)

yarn

All 4 resulted in the exact same crash.

user@WS-Q6JW6PN4MP 3.2.3-caret % yarn
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: @codertocat/hello-world-npm@npm:1.0.2::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Codertocat%2Fhello-world-npm%2F1.0.2%2F9384a68bd287b859f028683bcaacedd27803f1dd isn't supported by any available resolver
    at Bd.getResolverByDescriptor (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:391:1664)
    at Bd.getResolutionDependencies (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:391:1147)
    at TR.getResolutionDependencies (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:435:4101)
    at Bd.getResolutionDependencies (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:391:1176)
    at ce (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:439:7562)
    at /Users/user/.node/corepack/yarn/3.2.3/yarn.js:439:8135
    at async /Users/user/.node/corepack/yarn/3.2.3/yarn.js:439:7666
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 19ms

user@WS-Q6JW6PN4MP 4.0.0-rc-caret % yarn
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: @codertocat/hello-world-npm@npm:1.0.2::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Codertocat%2Fhello-world-npm%2F1.0.2%2F9384a68bd287b859f028683bcaacedd27803f1dd isn't supported by any available resolver
    at ed.getResolverByDescriptor (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:135:1698)
    at ed.getResolutionDependencies (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:135:1177)
    at Fx.getResolutionDependencies (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:190:11001)
    at ed.getResolutionDependencies (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:135:1206)
    at Ce (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:195:7688)
    at /Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:195:8489
    at async /Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:195:7826
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 19ms

Here are all 4 minirepos I created to reproduce this issue:

Yarn patch.zip

In all 4 cases, removing %3A%3A__archiveUrl=https%253A%252F%252Fnpm.pkg.github.com%252Fdownload%252F%2540Codertocat%252Fhello-world-npm%252F1.0.2%252F9384a68bd287b859f028683bcaacedd27803f1dd bit from resolution caused installation to proceed just fine & applied patch just fine.

I have a suspicion that releasing 1.0.3 version of test package would break patch on 3.2.3 as resolution created by patch-commit would no longer be applicable. But that's another story.

sargunv commented 1 year ago

I found this article quite helpful to understand the bug and how to work around it: https://charles-stover.medium.com/patching-packages-in-yarn-berry-72e4ded29a56

quisido commented 8 months ago

I'm still getting this in v4.1.0. I wish Yarn would implement some sort of e2e test around the yarn patch command to verify that it works without regression.

quisido commented 8 months ago

For documentation's sake, I fixed this by changing %3A%3A to :: before each __archiveUrl= and by moving #~/.yarn/patches/... to before the :: in each instance as well.

BasixKOR commented 8 months ago

Here's a simple reproduction I came up with:

const { writeFile } = require(`fs/promises`);

await writeFile('.yarnrc.yml', `
npmScopes:
  types:
    npmRegistryServer: https://registry.npmjs.com
`);

await packageJsonAndInstall({
  devDependencies: {
    "lodash": "^4.17.21",
    "@types/lodash": "^4.14.202"
  }
});

const { path } = JSON.parse(await yarn('patch', '@types/lodash', '--json'));
await yarn('patch-commit', '-s', path);
await expect(yarn('install')).resolves.toBeTruthy();

@lachlanhunt Could you add this into your issue description?

pniederlag commented 7 months ago

just being a fresh user of yarn I find 'yarn patch' not to be working (yarn 4.1.1 with custom npmRegistry, scoped packages and workspaces). Will now try to dig into it and find a workaround

pniederlag commented 7 months ago

just being a fresh user of yarn I find 'yarn patch' not to be working (yarn 4.1.1 with custom npmRegistry, scoped packages and workspaces). Will now try to dig into it and find a workaround

it seems to work when I do:

edit resolution field in yarn.lock, drop everything '::__archiveUrl.*'
start regular yarn patch, yarn patch-commit -s procedure
run yarn install

seems like it's doing as intended, while not being the most easy user experience though. ;)

I found some hints in #3012 regardingt the usage of __archiveUrl and non-conformant registry-servers (artifactory in my case), maybe that helps in getting this issue ironed out at some later point

gozegotushar commented 1 month ago

i am also facing same from last one week. not able to solve it.

gozegotushar commented 2 days ago

i was getting issue with installing yarn with homebrew.

it worked after updating my yarn & node version to : node : v20.8.1 yarn : 4.2.2

to install yarn please follow this link : https://stackoverflow.com/a/78307944/2382964

yarnpkg / berry