search

sourcegraph / scip-typescript

SCIP indexer for TypeScript and JavaScript
Apache License 2.0
44 stars 17 forks source link

Support PnP strategy in Yarn Berry #259

Open keynmol opened 1 year ago

keynmol commented 1 year ago

context

In some cases if you have "extends": "@package" in tsconfig.json, auto-indexer doesn't work, failing with, for example, whereas building normally works:

error TS6053: File '@grafana/tsconfig' not found.

error: no files got indexed. To fix this problem, make sure that the TypeScript projects ["/sources"] contain input files or reference other projects.

repro

Quickest is

  1. Clone https://github.com/grafana/grafana
  2. cd packages/grafana-ui && docker run -it -v (pwd):/sources -w /sources sourcegraph/scip-typescript:latest scip-typescript index

considerations

  1. It's not that extends doesn't work at all - the database for scip-typescript itself is perfectly indexable and it uses a shared config as well
valerybugakov commented 1 year ago

Yarn introduced a new installation strategy known as Plug'n'Play (PnP) starting from Yarn 2. It aims to address problems with the node_modules strategy used by traditional package managers like npm and Yarn 1.

The node_modules strategy works by creating a node_modules directory in your project root, where all your dependencies are installed. However, this can create problems because:

  1. The node_modules directory can become very large in size and contain many nested directories, leading to slow file system operations, long path names, and high disk space usage.
  2. There is no strict enforcement of package boundaries, so a package could mistakenly access another package's dependencies, leading to hard-to-diagnose bugs. Yarn PnP addresses these issues by storing all packages in a single .pnp.js file at the root of your project. This file acts as a map, pointing to the exact location of each package in a central cache directory managed by Yarn. This means no more node_modules directory in your project, leading to faster installations and less disk space usage. Because PnP changes how packages are stored and located, not all tools are compatible with it out of the box. They need to be able to understand the .pnp.js file to locate packages.

In our case, the TypeScript util parseJsonConfigFileContent tries to resolve the file @grafana/tsconfig/base.json using the traditional node_modules strategy, but it can't find it because Yarn PnP doesn't create a node_modules directory. To resolve this issue, we should use the PnP API provided by Yarn to resolve modules if PnP is enabled for the repo.

keynmol commented 1 year ago

Thanks @valerybugakov, that's some awesome investigative work.

More breadcrumbs - unmerged PnP PR for typescript: https://github.com/microsoft/TypeScript/pull/35206/files

Yarn claim that this PR is automerged by their plugin compat module: https://github.com/yarnpkg/berry/tree/master/packages/plugin-compat

Enabled by default, but perhaps it doesn't patch the particular method we use