ossf / allstar

GitHub App to set and enforce security policies
Apache License 2.0
1.22k stars 121 forks source link

[improvement] Allow app running with --once to exit with error when policy errors are encountered #503

Open twelsh-aw opened 3 months ago

twelsh-aw commented 3 months ago

We operate allstar inside a GHA with --once and noticed (due to #501) that a policy check would fail to apply at a certain point, but the app just exited with return code 0. It stopped monitoring for any more policies, but also didn't notify us of failure. The only way to notice is that the execution time of the GHA was much smaller than normal.

My initial thought was to update the goroutine to return the error, but realized this will also cancel the errgroup context and block any more app installations from running.

We could likely keep track of errors similar to how we keep track of failures, and do a check to return them after the fact. This can probably be opt-in behaviour if we are worried, but it might make sense to always be the behaviour for --once.

jeffmendoza commented 3 months ago

@twelsh-aw This sounds good. Allstar was originally meant to be run as a service only, but the --once usecase seems to be helpful to many folks.

I'm wondering if at some point, do we have two separate commands, but I don't think we are there yet.