search

lorenzwalthert / touchstone

Smart benchmarking of pull requests with statistical confidence
https://lorenzwalthert.github.io/touchstone
Other
53 stars 7 forks source link

if Benchmark fails, no failed check is visible or comment posted #105

Closed lorenzwalthert closed 2 years ago

lorenzwalthert commented 2 years ago

Now it’s silent, which is dangerous if the PR has big speed implications.

assignUser commented 2 years ago

I think the biggest "danger" here is if the comment job fails, as it is not listed in the pr-checks list. So everything would show up :heavy_check_mark:

lorenzwalthert commented 2 years ago

Yes, the two things in conjunction. The comment job should have an if: fails() clause that comments in the case of failure (hoping that that itself won't fail).

assignUser commented 2 years ago

We can create a job in the comment workflow that only runs if the receive flow failed.

lorenzwalthert commented 2 years ago

And what if the comment workflow fails for some other reason?

assignUser commented 2 years ago

mhm that's a bit of a "snake eating it's own tail"-situation... It would be best if we could turn the comment action into a pr check. That way it would be noticable if the action fails? I think this might be possible with the checks api... I'll look into it.

assignUser commented 2 years ago

Found a solution, pr incomming image image

assignUser commented 2 years ago

https://github.com/assignUser/simstudy/pull/20#partial-pull-merging