"Check could not be completed" despite all the rules are processed

[artem-shorokhov]

Hi there!

First a bit of context. We have a repository that holds the platform's configuration. The platform is multi-tenant and each tenant has its folder, like TEAM001 with non-production and production subfolders in the repository. With gitStream I mapped tenant's GitHub teams to each folder and implemented the following logic:

• if all the changes are related to the TEAM001/non-production folder and PR author is a member of respective GitHub team, PR is auto-approved;
• otherwise, the respective GitHub team is added as a required reviewer for every TEAMXXX folder that has changes.

Initial gitStream configuration

``` # -*- mode: yaml -*- manifest: version: 1.0 automations: add_private_topics_label: if: - {{ files | match(regex=r/private-topics/) | some }} run: - action: add-label@v1 args: label: "private-topics" color: "E94637" {% for client in clients %} add_system_label_{{ client.name | lower }}: if: - {{ files | match(regex=client.regex) | some }} run: - action: add-label@v1 args: label: {{ client.name }} approve_owned_nonprod_changes_{{ client.name | lower }}: if: - {{ files | match(regex=client.regex) | every }} - {{ files | match(regex=r/\/production\//) | nope }} - {{ pr.author_teams | match(term=client.team) | some }} run: - action: approve@v1 - action: add-label@v1 args: label: "auto-approved" color: "36A853" require_review_of_owned_prod_changes_{{ client.name | lower }}: if: - {{ files | filter(regex=client.regex) | match(regex=r/\/production\//) | some }} - {{ pr.author_teams | match(term=client.team) | some }} run: - action: add-reviewers@v1 args: team_reviewers: [{{ client.team | lower | replace(" - ", "-") | replace(" ", "-") }}] - action: require-reviewers@v1 args: team_reviewers: [{{ client.team | lower | replace(" - ", "-") | replace(" ", "-") }}] require_review_by_owners_{{ client.name | lower }}: if: - {{ files | match(regex=client.regex) | some }} - {{ pr.author_teams | match(term=client.team) | nope }} run: - action: add-reviewers@v1 args: team_reviewers: [{{ client.team | lower | replace(" - ", "-") | replace(" ", "-") }}] - action: require-reviewers@v1 args: team_reviewers: [{{ client.team | lower | replace(" - ", "-") | replace(" ", "-") }}] {% endfor %} clients: - name: "TEAM001" regex: "r/^TEAM001\//" team: "team-001" ... - name: "TEAM150" regex: "r/^TEAM150\//" team: "team-150" ```

So, we had 150 teams and 601 automations, main gitStream.cm check was running for 8-9 min and worked completely fine until we added team number 151 on the 9th of October, and it started failing after around 11 min with Check could not be completed and no steps being executed.

I assume that probably was not because of the addition of another few rules, but probably related to some changes on the gitStream side that affected performance.

I tried to optimize the configuration focusing on reducing the number of rules and ended up with the following:

Optimized gitStream configuration

``` # -*- mode: yaml -*- manifest: version: 1.0 automations: add_private_topics_label: if: - {{ files | match(regex=r/private-topics/) | some }} run: - action: add-label@v1 args: label: "private-topics" color: "E94637" {% for client in clients %} approve_owned_nonprod_changes_{{ client.name | lower }}: if: - {{ ( files | match(regex=client.regex) | every ) and ( files | match(regex=r/\/production\//) | nope ) and ( pr.author_teams | match(term=client.team) | some ) }} run: - action: add-label@v1 args: label: {{ client.name }} - action: approve@v1 - action: add-label@v1 args: label: "auto-approved" color: "36A853" require_review_{{ client.name | lower }}: if: - {{ not ( ( files | match(regex=client.regex) | every ) and ( files | match(regex=r/\/production\//) | nope ) and ( pr.author_teams | match(term=client.team) | some ) ) }} - {{ files | match(regex=client.regex) | some }} run: - action: add-label@v1 args: label: {{ client.name }} - action: add-reviewers@v1 args: team_reviewers: [{{ client.team | lower | replace(" - ", "-") | replace(" ", "-") }}] - action: require-reviewers@v1 args: team_reviewers: [{{ client.team | lower | replace(" - ", "-") | replace(" ", "-") }}] {% endfor %} clients: - name: "TEAM001" regex: "r/^TEAM001\//" team: "team-001" ... - name: "TEAM157" regex: "r/^TEAM157\//" team: "team-157" ```

This partially solved the problem as all the steps (add-label, add-reviewers, require-reviewers, etc.) are now executed properly, but the main gitStream.cm check is still failing after around 11 min with Check could not be completed:

I assumed that rules are processed alphabetically and tested the last automation in the loop for the last client in the list. This rule was processed after around 3 min:

This feels like all the automations are being processed within around 4 min, but the main check continues running until dies by timeout.