search

opensource-observer / oso

Measuring the impact of open source software
https://opensource.observer
Apache License 2.0
49 stars 13 forks source link

Setup alerts for broken data pipelines #1070

Open ravenac95 opened 3 months ago

ravenac95 commented 3 months ago

Describe the feature you'd like to request

We currently don't have any alerts for any of our pipelines failing other than manually checking right now. We should have automated checks for ourselves.

Describe the solution you'd like

Some kind of automated check messages us either on discord or some other method.

Describe alternatives you've considered

N/A

ryscheng commented 2 months ago

Dagster+ solution https://docs.dagster.io/dagster-plus/managing-deployments/alerts#dagster-alerts

Or the cheap version is to periodically scan Dagster API for failures, then throw that into a Discord comment

ravenac95 commented 1 month ago

We can also think about doing things like mentioned here: https://github.com/dagster-io/dagster/discussions/19944

My thought would be something like this but it would send a message to our discord.

In order to have that setup we'd need to connect the secret to our dagster deployment. I would then pull that secret directly from the python code (perhaps we have an environment variable to reference it's location). Otherwise we can use the already provided secret resovler: https://github.com/doitintl/kube-secrets-init that is a side car to our applications. However, I think it would be better to have our dagster hand written components use the secret manager directly as opposed to relying on the side car. It just exposes less secrets in the environment directly. So you can't accidentally run env and log secrets by accident.