Using dependency graph to reduce file scanning

[t-ayola]

Hello, we are experiencing very long debugging times in the time it takes to reach a breakpoint. Even with scoping down the scanned files with the outfile argument

We work in a large yarn v1 monorepo with a sparse dependency graph. We'd like to use the dependency graph to restrict the amount of file scanning js-debug needs to do on startup. With the arrival of npm workspaces, it feels reasonable for js-debug to be able to reason about workspaces in a monorepo

[connor4312]

It sounds like the right current solution is restricting your outFiles.

If I understand your proposal, you're saying that we would scan dependencies of the program entrypoint to discover sourcemaps. At first glance, I'm not too big a fan of this.

• Currently discovering sourcemaps is very fast since we only need to find filenames that match a given pattern. Crawling the dependency tree would involves reading every file, parsing its AST, and figuring out its imports and exports. I would only expect to see benefits in staggeringly large repositories where the number of irrelevant files vastly outnumbers the relevant ones. For these cases, we already have the outFiles, and configuring those is going to lead to much faster scanning than AST parsing.
• The dependencies of a program cannot always be known without first evaluating the program. For example, it's very common to glob for test files and dynamically require or import them in a test runner script.
• The use of more exotic loaders and resolution utils would likewise befuddle a AST-based approach.
• If the user runs some runtime or dialect that the ecmascript parser used in js-debug cannot parse, we may also fail to parse its dependencies.

[connor4312]

Updating from Teams discussion:

@t-ayola:

To give a bit more context here, we're working in a large yarn v1 monorepo, and while we've configured outfiles to be as narrow as possible (while still providing a single debug profile that works for the whole repo), we're still seeing scan times of up to minutes.

The proposal was to be able to use the monorepo structure to restrict the outfiles even more. Today, we have to scan the "lib" folders of all packages, no matter which one we're debugging, but really we only care about the lib folder of our current package and its dependents in the repo. So this was one thought we had for a way to make js-debug smarter about how it can manage scanning in monorepo setups.

@connor4312:

Ah, so you mean literally scanning package.json dependencies? ... I think that's sensible. Not terribly hard, just some transversal. I would probably implement it as a scanning of npm workspaces, which you appear to be using. I think it would be safe to do it automatically if the outFiles are not user-configured.

I'm unsure whether I'd have time this iteration (month) to get to it, but would welcome a PR. Here's a code pointer to the place where we do the automatic "guessing" of outFiles today https://github.com/microsoft/vscode-js-debug/blob/8fa24a71b84043a3c7065e4c64e1a9541ec518b2/src/ui/configuration/nodeDebugConfigurationResolver.ts#L249