Report failures of periodic jobs to the cluster-api Slack channel

[sbueringer]

I noticed that CAPO is reporting periodic test failures to Slack, e.g.: https://kubernetes.slack.com/archives/CFKJB65G9/p1713540048571589

I think think this is a great way to surface issues with CI (and also folks can directly start a thread based on a Slack comment like this)

This could be configured ~ like this: https://github.com/kubernetes/test-infra/blob/5d7e1db75dce28537ba5f17476882869d1b94b0a/config/jobs/kubernetes-sigs/cluster-api-provider-openstack/cluster-api-provider-openstack-periodics.yaml#L48-L55

What do you think?

[sbueringer]

cc @chrischdi @fabriziopandini

[k8s-ci-robot]

This issue is currently awaiting triage.

CAPI contributors will take a look as soon as possible, apply one of the triage/* labels and provide further guidance.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[chrischdi]

Oh wow, yeah that would be a great thing. I just fear that it may pollute the channel too much. But we could try and fail fast by asking for feedback if it is too much later on in the community meeting or via a slack thread/poll.

[killianmuldoon]

do we know if this respects testgrid-num-failures-to-alert? If so it could be great.

[sbueringer]

I'm not sure if it respects that. We could try and rollback if it doesn't?

[sbueringer]

If it still pollutes the channel too much after considering testgrid-num-failures-to-alert we have to focus more on CI :D

(I"m currently guessing that we would get one Slack message for every mail that we get today, but I don't know)

[killianmuldoon]

One slack message per mail would be perfect - more would disrupt the channel

WDYT about enabling it for CAPV first?

[killianmuldoon]

Also fine with making the change and rolling back if it doesn't work

[sbueringer]

One slack message per mail would be perfect - more would disrupt the channel WDYT about enabling it for CAPV first?

Fine for me, we can also ask the OpenStack folks how spamy it is for them today (cc @mdbooth @lentzi90)

[lentzi90]

For CAPO we get a message for every failure and email only after 2 failures in a row. I think it has been tolerable for us, but that indicates it does not check testgrid-num-failures-to-alert (at least the way we have it configured)

[sbueringer]

Hm okay, every failure is just too much. So we should probably take a closer look at the configuration / implementation. One message for every failure just doesn't make sense for the amount of tests/failures we have (the signal/noise ratio is just wrong)

[fabriziopandini]

+1 to test this if we find a config reasonably noisy (but not too much noisy) cc @kubernetes-sigs/cluster-api-release-team

/priority backlog /kind feature

[adilGhaffarDev]

+1 from my side too. Tagging CI lead @Sunnatillo I will add this to improvement tasks for v1.8 cycle. CI team can look into this one.

[Sunnatillo]

Sounds great. I will take a look

[Sunnatillo]

I guess this testgrid-num-failures-to-alert should help with the amount of the noise. If we set it, for example, to 5 we will be sure that we will receive messages about constantly failing tests. This makes the config to sent the alert after 5 continuous failures.

[Sunnatillo]

/assign @Sunnatillo

[lentzi90]

@Sunnatillo testgrid-num-failures-to-alert does not affect the slack messages for CAPO at least. Only emails are affected by that in my experience.

[Sunnatillo]

@Sunnatillo testgrid-num-failures-to-alert does not affect the slack messages for CAPO at least. Only emails are affected by that in my experience.

Thank you for the update. I will open the issue in test-infra, try to find the way to do it.

[Sunnatillo]

I opened an issue regarding this in test-infra: https://github.com/kubernetes/test-infra/issues/32687