wedamija
commented
8 months ago The only other thing is that I assume the only way this can happen is via some kind of transaction rollback situation. Like:
- Event that would create group comes in, increments number, saves group, somehow project counter is rolled back (doesn't seem plausible since in transaction?)
- 2 events that would create group A & B come in. Due to some race condition, one commits later, or one rolls back, and somehow this breaks the counter (I don't have a solid mechanism for how this would happen)
- Something to do with merging/unmerging groups. We increment the counter on unmerge I believe, so this seems like a reasonable place to look as well. Maybe we don't handle transactions right there?
lobsterkatie
About a month ago, we started seeing a bunch of
IntegrityErrors in thestore.save_eventtask. More specifically, they wereUniqueViolation('duplicate key value violates unique constraint "sentry_groupedmessage_project_id_<random internal hexcode>_uniq"\nDETAIL: Key (project_id, short_id)=(12311121, 41598) already exists.errors. (Legal folks, don't worry - those aren't real numbers, they're my dogs' birthdays and adoption dates.)@wedamija eventually tracked it down to the projects'
CounterandGroup.short_idvalues being out of sync. Each project has one row in thesentry_projectcountertable, whose job it is to keep track of the most recent/highestshort_idassigned to an issue in that project, such that a new group coming in can take theshort_idone higher than that, and never run into a conflict. (We have a unique constraint on theproject_id/short_idpairing in thesentry_groupedmessagetable.) Normally, that works great.For each project throwing the
UniqueViolationerrors, however, the project counter value had fallen one behind the highest assignedshort_id. Therefore, every time we tried to create a new issue for that project, we tried to give it what the project counter thought was the next availableshort_id, only to find that thatshort_idwas already taken by the most recently created existing issue. At that point we'd just error out, so the existing newestshort_idwould stay newest, the project counter wouldn't get incremented, and the group wouldn't get created. Then the next novel event would come in, we'd try to create a group for it, all the same things would happen, and we'd be effectively stuck. Projects in this state can still accept events into existing issues, but can't create new any ones.Steps to resolve this:
[x] Manually fix broken projects by forcing the counter to have the right value. When this was a one-off, we figured it was gremlins and hoped it never came back. Then it did, so we fixed those projects, and then... it was clear we needed a better solution. PRs for those manual fixes:
[x] Add a workaround, so that any project found to be stuck is immediately unstuck and no new groups are dropped. PRs for that:
[ ] Figure out how common this is, using this DD query and this GCP logs query. (NOTE: You may have to update the timeframes on each of these, as I can't seem to get them to cover a generic timeframe which will update itself.)
[ ] Figure out how the heck our data is getting out of whack in the first place.
Two ideas from @wedamija:
Simplify the code we use to increment the project counter, so that it's a simple update. This would require creating a
Counterrecord for each project when it's created, and backfilling any projects which currently don't have aCounter.The timing of these errors appearing lines up pretty well with our upgrade to Django 5. See if we can figure out anything that changed there which might be causing this.