[rush] Extend incremental build ignore capabilities (per command, by monorepo/workspace devDependencies)

[stekycz]

Summary

We have various custom commands leveraging the build cache. However, some of those commands do not actually depend on all devDependencies in package.json. I.e. when a dev dependency (e.g. another project in the monorepo) changes, all commands cache is invalidated even though we know it is not relevant for some of those commands.

We can ignore specific files in each project to be considered ignored for changes. However, we cannot do that for monorepo projects in dependencies. I believe it should be considered for devDependencies only though.

A small detail relevant for both - ignoring files and ignoring dependencies - is that the current file ignore works for all commands. It would be nice to have the possibility ignore files (and devDependencies) independently for each command (but keeping the common configuration for all commands).

E.g. rush-project.json

{
  "incrementalBuildIgnoredGlobs": [
    "**/*.md"
  ],

  "incrementalBuildIgnoredDevDependencies": [
    "@rushstack/foo"
  ],

  "operationSettings": [
    ...
    {
      "operationName": "foo",
      ...
      "incrementalBuildIgnoredGlobs": [
        "**/*.md", // Notice this is duplicated and not inherited. Inheritance would make it complicated to make an exception including generally ignored files
        "**/*.json"
      ],

      "incrementalBuildIgnoredDevDependencies": [
        "@rushstack/bar", // Notice the dependency `@rushstack/foo` is not ignored here
      ],
    },
    ...
  ]
}

[dmichon-msft]

I am very much in favor of moving all settings related to tracking of files/dependencies into the operationSettings structure, because that is necessary for dynamic configuration of phases. I'm less sure that filtering out certain devDependencies is the best way to do the filtering of dependencies on phases in other projects, however.

The motivating example for me has to do with splitting transpilation from type checking, in which scenario all runtime dependencies will need to be ignored and the only dev dependencies that should be depended on are rigs and (potentially) local Heft plugins.

Essentially, while I think the capability to ignore certain dev dependencies has some utility, we probably want to address this configurability in a much broader fashion.

Separately there is a concern that, absent a tool like strace that can be used to validate that the rush-project.json configuration is strictly correct (i.e. does not read any ignored files, does not write to any output files), reducing the set of projects/files included in build cache key computations is dangerous. A bad cache entry being written is much more consequential than a redundant cache miss.

[stekycz]

The use case I am for is an integration test. E.g. I have another project in the monorepo (let's call it server) and I want to write a few integrations tests to validate my service can properly communicate with the server. I.e. I want to start the server and then call function in my service from my integrations tests. However, I do not want to rebuild my code, rerun unit tests or invalidate any other cached result when the server changes. I just want to execute integration tests in order to prevent regression.

You can admin I should use contract based testing and you would be right. However, that is a long shot in my specific case. I already have integration tests and (unfortunately) almost no machine processable documentation for contract testing. Before I get there, I would like to leverage Rush.js feature effectively.

Anyway, I agree a much broader fashion refactoring of configurability would be great. I can actually help a bit, e.g. writing tests, implementing some parts.

[dmichon-msft]

The main challenge is that this will need some refactoring changes to the way build cache hashes are computed. I keep trying to start the process of refactoring the build cache ids to mirror the operation execution graph, then get stalled because the resulting PR is huge and hard to review.

See, e.g. #3696, which was my last attempt at doing this.

[dmichon-msft]

For more details on my above comment, the main change that needs to happen is that the build cache keys need to be a function of the build cache keys of the operations they depend on (or at least the sum total data that went into said cache keys), combined with any local file dependencies and external config settings (e.g. environment variables)

[stekycz]

I see, there really is a lot of changes. In understand what needs to happen and I think of what steps might go there instead of creating one big step. I will post a comment here and create a PR when I get an idea.