x/build/cmd/watchflakes: opens duplicate issues

[dmitshur]

watchflakes is opening many duplicate issues recently, like "runtime:cpu4: TestUsingVDSO failures" is #67392 and #67391 and others.

I suspect the problem is that they're not added to the Test Flakes project successfully, so watchflakes doesn't realize it already filed an existing issue.

CC @golang/release.

[dmitshur]

Logs show it's running into a limit of items that can be added to a GitHub project:

# new issue: runtime:cpu4: TestUsingVDSO failures
2024-05-15 13:52 gotip-linux-ppc64_power8 go@6ccd8e4c runtime:cpu4.TestUsingVDSO
https://ci.chromium.org/b/8747866791326770913
pkg == "runtime:cpu4" && test == "TestUsingVDSO"

=== RUN   TestUsingVDSO
=== PAUSE TestUsingVDSO
=== CONT  TestUsingVDSO
    vdso_test.go:51: GO_WANT_HELPER_PROCESS=1 /bin/strace -f -e clock_gettime /home/swarming/.swarming/w/ir/x/t/go-build3885972143/b001/runtime.test -test.run=^TestUsingVDSO$
    vdso_test.go:57: /bin/strace: Process 140051 attached
        /bin/strace: Process 140052 attached
        /bin/strace: Process 140050 attached
        /bin/strace: Process 140053 attached
        [pid 140050] ???( <unfinished ...>
        [pid 140049] --- stopped by SIGURG ---
        /bin/strace: ptrace(PTRACE_LISTEN,pid:140049,sig:0): Input/output error
        /bin/strace: Process 140049 detached
        /bin/strace: Process 140051 detached
        /bin/strace: Process 140052 detached
        /bin/strace: Process 140050 detached
        /bin/strace: Process 140053 detached
    vdso_test.go:70: exit status 1
--- FAIL: TestUsingVDSO (0.14s)

watchflakes: 2024/05/15 16:10:33 1
watchflakes: 2024/05/15 16:10:33 2        mutation($Project: ID!, $Issue: ID!) {
watchflakes: 2024/05/15 16:10:33 3          addProjectV2ItemById(input: {projectId: $Project, contentId: $Issue}) {
watchflakes: 2024/05/15 16:10:33 4            clientMutationId
watchflakes: 2024/05/15 16:10:33 5          }
watchflakes: 2024/05/15 16:10:33 6        }
watchflakes: 2024/05/15 16:10:33 7      
watchflakes: 2024/05/15 16:10:33 graphql error: Projects cannot have more than 1200 items. To add more, please archive or delete existing items.

[dmitshur]

I've stopped it from running as a short-term workaround. Removing Soon label.

We've applied to https://github.blog/changelog/2024-02-12-github-issues-projects-projects-without-limits-private-beta/, which should help.

In the meantime, I'll work on a change to make it handle this limit more gracefully (i.e., don't keep opening new issues without end).

[dmitshur]

We've made two changes to help with this:

• there's now an auto-archive workflow in the Test Flakes project
  • it archives closed issues after 4 weeks of inactivity; we can adjust the exact time to strike the optimal balance between it reopening existing issue vs opening a new one
• we've started taking advantage of GitHub's early support for removing limits on project issue count

That should prevent this from happening again. After archiving closed issues that hadn't been updated in the last 4 weeks, there's now 519 items in the Test Flakes project (down from 1200).

I've resumed watchflakes and it's working okay now. Closing this as resolved.