travjenkins
commented
2 years ago This will be a good task for post-mvp. This kind of functionality will be what the UI is targeting when we do some refactoring/developing new patterns based on the learning thus far.
Note: this is intended to be a future enhancement that we shouldn't do until we are more certain that shard failures are not caused by something we've done vs something the user has done.
Problem We've added the ability to see if your workflows are running or not through the UI, but it would be awesome to get notifications when something happens to them. I for instance have a lot of my captures that are happily humming along, but my SQL ones have failed and I would have liked to have known when that happened.
Proposed Solution Add a boolean to the users grid that a user can select -- only for themselves -- titled notification that has the values "on" or "off". We should keep it simple for now.
When the value is "On", we should send them an email when any shards that they originally published goes from "Active" to any state that's not running. The email should say the following
"Your (capture or materialization) ${name} has ${what happened} as of ${human-readable-timestamp}. Here are logs related to the failure:
${logs}"
Because we re-start tasks every 5 minutes, we should probably do something clever around that like only send the notification once per day per shard-failure reason