search

vipyrsec / dragonfly-mainframe

The mainframe for Dragonfly
https://docs.vipyrsec.com/dragonfly-mainframe/
MIT License
5 stars 1 forks source link

Occasional UniqueViolation in `POST /batch/package` #233

Open jonathan-d-zhang opened 6 months ago

jonathan-d-zhang commented 6 months ago

As far as I can tell, the package deduplication is not correctly deduplicating. I don't know why.

jonathan-d-zhang commented 6 months ago

i ruled out concurrency issues, since we only run 1 loader, which hits the batch queue endpoint once per minute, and the endpoint doesn't take over a minute to run so it wouldn't overlap. there is also no interaction between other endpoints: even the strictest concurrency setting we use, SELECT .. FOR UPDATE, wouldn't stop us from reading the row that it locked