[Bug?]: Dependencies don't always point to their virtual instances in `yarn info`

pauldraper commented 2 years ago

Self-service

[ ] I'd be willing to implement a fix

Describe the bug

yarn info --virtuals doesn't show the dependencies of virtual packages.

To reproduce

package.json

{
  "dependencies": {
    "webpack": "5.66.0"
  }
}

Run

yarn install
yarn info --virtuals webpack

The output shows

├─ webpack@npm:5.66.0
│  ├─ Instances: 1
│  ├─ Version: 5.66.0
│  │
│  ├─ Exported Binaries
│  │  └─ webpack
│  │
│  └─ Dependencies
...
│     ├─ acorn-import-assertions@npm:^1.7.6 → npm:1.8.0
│     ├─ acorn@npm:^8.4.1 → npm:8.7.0
...
│
└─ webpack@npm:5.66.0 [4c3f2]
   ├─ Version: 5.66.0
   │
   └─ Peer dependencies
      ├─ @types/webpack-cli@* → ✘
      └─ webpack-cli@* → ✘

Notice that the virtual package (webpack@npm:5.66.0 [4c3f2]) shows no dependencies.

I expected it to have a dependency on several packages, including acorn@npm:8.7.0 and acorn-import-assertions@npm:1.8.0 [6d128].

Environment

System:
  OS: Linux 5.4 Ubuntu 20.04.3 LTS (Focal Fossa)
  CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Binaries:
  Node: 16.10.0 - /tmp/xfs-feeee10b/node
  Yarn: 3.1.1 - /tmp/xfs-feeee10b/yarn
  npm: 7.24.0 - ~/.nvm/versions/node/v16.10.0/bin/npm

Additional context

In other contexts, the virtual package has the expected dependencies.

For example, going the reverse direction, acorn@npm:8.7.0 is a dependent of webpack@npm:5.66.0 [4c3f2].

yarn info -R --dependents --virtuals acorn

└─ acorn@npm:8.7.0
   ├─ Version: 8.7.0
   │
   ├─ Exported Binaries
   │  └─ acorn
   │
   └─ Dependents
      ├─ acorn-import-assertions@npm:1.8.0 [6d128]
      ├─ webpack@npm:5.66.0
      └─ webpack@npm:5.66.0 [4c3f2]

Using why shows the same:

yarn why -R acorn
└─ test50@workspace:.
   └─ webpack@npm:5.66.0 [4c3f2] (via npm:5.66.0 [4c3f2])
      └─ acorn@npm:8.7.0 (via npm:^8.4.1)

pauldraper commented 2 years ago

Side note: My goal here is to get a comprehensive list of the resolved package dependency graph for analysis.

If there's a better way to do this, I'd love to know.

arcanis commented 2 years ago

It shows dependencies as part of a non-virtual base record. Since those dependencies will never change regardless of the virtual, duplicating them on each virtual record would take a very large amount of vertical space for nothing.

pauldraper commented 2 years ago

a very large amount of vertical space for nothing.

They are only printed if --virtuals is explicitly chosen.

Since those dependencies will never change regardless of the virtual

Isn't the entire point of virtual packages is that the (peer) dependencies can change?

Virtual package A1 depends on a virtual package B1, yet yarn info only shows that virtual package A1 depends on concrete package B. That loses information, because B and B1 are not the same package.

That's a bit abstract. Consider the above example, to show why virtuals deps are necessary.

In reality,

webpack@npm:5.66.0 [4c3f2] depends on acorn-import-assertions@npm:1.8.0 [6d128]
acorn-import-assertions@npm:1.8.0 [6d128] depends on acorn@npm:8.7.0.

But yarn info won't reveal that. It only says

webpack@npm:5.66.0 [4c3f2] exists
webpack@npm:5.66.0 depends on acorn-import-assertions@npm:^1.7.6
acorn-import-assertions@npm:1.8.0 has no dependencies (!!)

Only virtual packages of acorn-import-assertions depend on acorn. But yarn info fails to tell us which virtual package of acorn-import-assertions is used by webpack, because it hides the deps of the webpack virtual package.

(Again, the idea is to obtain a dependency graph, so that I can consume it with analysis tools.)

arcanis commented 2 years ago

Isn't the entire point of virtual packages is that the (peer) dependencies can change?

The peerDependencies resolutions can change (and those are the ones you see in the Peer dependencies section from your paste). The dependencies resolutions are static and cannot change from one virtual to the other, hence why they are factored in their own section.

Virtual package A1 depends on a virtual package B1, yet yarn info only shows that virtual package A1 depends on concrete package B. That loses information, because B and B1 are not the same package.

Right, that's the bug. Yarn does show the dependencies, it's just that it shows the base package rather than the virtual ones 🙂

This is because the base package data structure, used to print the regular dependencies, is never mutated during the virtual tree post-processing (only its virtual instances are) and thus if one of its dependencies get virtualized it'll never know. We'll need to:

Tweak the sortMap call to guarantee that we traverse base packages before their virtual instances
When traversing a package's virtual instance, update the registered dependencies of its base package

pauldraper commented 2 years ago

Thanks for the thoughts. You're certainly familiar, though from my poking yarn why it seems that there are virtual -> virtual dependencies that aren't in the base. But I might be wrong about that.

yarnpkg / berry