Add "automatic workspace discovery" to `transformChanges`

[mrnugget]

(Just replicating what I wrote in Slack here so we don't lose it)

While working on this [the "dynamic workspace discovery"] I’ve grown convinced that this “dynamic folder location discovery by using sourcegraph search” could/should also be used for transformChanges. In addition to directory: we could allow rootAtLocationOf and use the same mechanism to auto-discover the folders into which changes should be grouped.)

In more concrete terms: in addition to having this

transformChanges:
  group:
    - directory: client
      branch: hello-world-client # will replace the `branch` in the `changesetTemplate`
    - directory: docker-images
      repository: github.com/sourcegraph/sourcegraph
      branch: hello-world-infra

changesetTemplate:
  branch: hello-world # This branch is the default branch and will be overwritten for each additional changeset.
  # [...]

we could have something like this:

transformChanges:
  group:
    # Determine the directory locations using Sourcegraph search
    - rootAtLocationOf: package.json
      # Allow globbing when specifying the repository name
      in: github.com/sourcegraph/automation*
    - rootAtLocationOf: README.md
      # Stay backwards compatible:
      repository: github.com/sourcegraph/sourcegraph

changesetTemplate:
  # And we also use templating:
  branch: hello-world-${{ transformChanges.group.directory}}
  # Could also look like this:
  branch: hello-world-${{ changes.directory}}
  # [...]

[malomarrec]

+1, adding one more use case here user: +1 for the problem and solution proposed here from https://github.com/sourcegraph/accounts/issues/6859

Use case

In a monorepo structured by package

app
--- x
   |--- package-info.java
   |--- y2
         |---package-info.java
   |--- y1 
         |---package-info.java

• Some, but not all subpackages have package-info.java .
• The spec should adjust to the repo structure, and the user shouldn't have to change the repo or add files to use batch changes.
• The user wants to create changesets at for a specific path level com.app.X, but not com.app.X.Y because that creates too many tiny changesets

Current behaviour

• Workspaces: rootAtLocationOf is not flexible enough, there is a package-info.java in all subpackages, which creates too many changesets
• transformChanges: doesn't work because it requires listing out all changesets

Proposed behaviour

• Add glob syntax to directory

All subdirectories of app of depth 1

  group:
    - directory: app/*

changesetTemplate:
   branch: batch-changes/-${{ replace steps.path "/" "-" }} # generate unique branch name with path

All subdirectories of app of depth 2

  group:
    - directory: app/*/*

changesetTemplate:
   branch: batch-changes/-${{ replace steps.path "/" "-" }} # generate unique branch name with path

[kkom]

Thanks @malomarrec!

Add glob syntax to directory

Another idea here – perhaps a regex syntax with capture groups could be more flexible, but still clear?

The idea would be to just search for files / directories, using a regex with some capture groups. Every workspace would be defined by the tuple of captured values.

Example: /root-project/(lib-data-.*)/src/(module-.*) to generate one workspace per module of a "data" library.

We don't have a need for something like this now, but it occurred to me as a possibility. I think this approach is very flexible, but still not overcomplicated.

[NickChoiRBX]

Just wanted to say that for our csharp codebase it would be enormously helpful to target something like the <directory-path>\*.csproj. In csharp, there isn't an obvious single filename I can think of that I could target for every project.

[malomarrec]

@NickChoiRBX thanks for the feedback here. If you had to write a script to define what a project is, what would that be (feel free to DM me or reply here)!

[malomarrec]

A monorepo use case from https://github.com/sourcegraph/accounts/issues/6859

There is a repo with Terraform files. The user runs sed commands to upgrade to the next Terraform version.

Workspaces are grouped by cloud account, then by service

aws-acct-1
    project-1
    project-2
    project-3
aws-acct-2
    project-1
    project-2
    project-3
gcp-acct-1
    project-1
    project-2
azure-acct-1
    project-1
    project-2

A workspace is azure-acct-1/project-1 for example. The use case is:

• Run steps in all subrepos of gcp and aws, not Azure.
• From the output, create a changeset per cloud account (eg. gcp-acct-1), grouping all diffs from sub-projects into a single changeset. Without the grouping, there is one PR for each of 100s of services in each account: that is impossible to manage.

The ideal situation would be:

(aws|gcp)-.*/.*.

An acceptable situation would to just list out glob type path:

-aws-acct-1/*
-aws-acct-2/*

[malomarrec]

Subsumed into #28246 and RFC 574 WIP: Make Batch Changes work for common monorepo structures