EmerisHQ / demeris-backend

Monorepo containing all the Demeris backend code and infrastructure definitions.
GNU Affero General Public License v3.0
8 stars 1 forks source link

Add alerts for tracelistener database errors #795

Open Pitasi opened 2 years ago

Pitasi commented 2 years ago

Use this query as a base:

{container="trace-store"} |= "database error" != "affected rows are zero"

(we currently log quite a log of "affected rows are zero" but we can ignore them, in the future we won't have these logs).

These kind of errors are really bad. They mean we couldn't write to database so we have an inconsistent state, usually manifested as wrong balances for users. We should be notified early instead of waiting for users to complain.

To recover from this state we need a bulk import (and maybe to fix a bug in the Tracelistener).

DoD: