Feature request: Allow specifying a webhook to receive Plugin errors

[beaugunderson]

In working with Reagan yesterday and talking to Hines just now I think it would be very helpful for production plugin users to be able to receive errors reports for their plugins at a webhook URL of their choosing. This might serve a different purpose than the syslog logging we've talked about but it would be structured data (JSON) instead of text tracebacks, and so it may be worth implementing for that reason (maybe higher priority than the syslog logging even?)

Curious as to your thoughts...

[andhines]

Should also use sentry to improve observability, don't rely only on logs

[beaugunderson]

Should also use sentry to improve observability, don't rely only on logs

For context, this is important for us internally--currently workflow-kit protocols do capture exceptions in Sentry but they all go into the same bucket; I added a PR this morning to move them to separate buckets with the same tag. I think we should do the same thing for exceptions that occur during plugin handling (again, for internal use).

[aduane]

How do you see this being implemented? An ERRORS_WEBHOOK_URL set in the container that, if set, POSTs the JSON error payload to that URL? Or that the plugin runner can emit events that a plugin can be written to receive and handle (and presumably errors occurring in such an error handling plugin would not emit the error event to avoid an error ouroboros)?