Ability to skip tasks on irrelevant file changes

boschni commented 1 year ago

Which project is this feature idea for?

Turborepo

Describe the feature you'd like to request

I would like to be able to skip tasks when the inputs did not change.

For example, given the following workspace:

src/index.ts
src/index.spec.ts
README.md

And the following turbo configuration:

{
  "$schema": "https://turbo.build/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "inputs": ["src/**/*", "!src/**/*.spec.ts"],
      "outputs": ["dist/**"]
    },
    "test": {
      "dependsOn": ["^build"],
      "inputs": ["src/**/*"]
    }
  }
}

Then in a CI pipeline I want to be able to:

Only run the build task when index.ts changed, or any inputs from the tasks specified in dependsOn.
Only run the test task when index.ts or index.spec.ts changed, or any inputs from the tasks specified in dependsOn.

Currently, a commit that only modifies README.md will cause turbo run <task> --filter=...[HEAD^1] to always run the given task even though this might not be needed.

Describe the solution you'd like

A possible solution would be to:

Add a CLI command like --filter-inputs to filter by the inputs of the given tasks.

Running turbo run build --filter=...[HEAD^1] --filter-inputs would only include index.ts.

Running turbo run test --filter=...[HEAD^1] --filter-inputs would only include index.ts and index.spec.ts.

Running turbo run build test --filter=...[HEAD^1] --filter-inputs would be less useful as it would include index.ts and index.spec.ts for both tasks. To support multiple tasks turbo would need to be be able to filter individual tasks instead of files but I'm not sure if this aligns well with how turbo currently works.

Describe alternatives you've considered

It is possible to ignore files with turbo run <task> --filter=...[HEAD^1] --ignore "**/*.md" but this does not scale as each workspace and task might have different requirements.

nathanhammond commented 1 year ago

Realistically this is a bug. We take too-conservative of an approach in calculating whether or not a package has changed.

In particular, getChangedFiles should be taking into account the inputs for each of the tasks here: https://github.com/vercel/turbo/blob/main/cli/internal/scope/scope.go#L363-L380

We capture the need to make this change here: https://github.com/vercel/turbo/blob/main/cli/internal/scope/scope.go#L212-L213

jonaslindstr commented 2 months ago

I am able to reproduce this bug using the example repo and latest canary.

run npx create-turbo@v2.0.13-canary.0
commit the lock file
add "inputs": ["$TURBO_DEFAULT$", "!README.md"] to the lint task in turbo.json (exactly like in the docs), commit
create packages/ui/README.md and commit
npm run lint -- -F='[HEAD^1]' -> runs lint task in the ui package, even though only README.md has changed

Running with dry-run=json: README.md is not part of inputs (correct) Running with -vvv: globwalk includes the README file in each package rather than excluding it. I haven't dived into the source code but you get the feeling that it should be the opposite.

vercel / turborepo