Zero downtime tracelistener bulk imports - Githubissues

EmerisHQ / demeris-backend

Monorepo containing all the Demeris backend code and infrastructure definitions.

GNU Affero General Public License v3.0

8 stars 1 forks source link

Zero downtime tracelistener bulk imports #784

Open Pitasi opened 2 years ago

Pitasi commented 2 years ago

DoD:

[ ] make bulk import use Upsert queries version instead of Insert (we could add a flag to decide, because plain INSERTs runs much faster in cockroach)
[ ] after the bulk import is done here, run a logic that deletes all rows from all the known table that have the height column lower than the bulk import height
[ ] add new unit tests to test corner cases (newer data already present in db, or older data present in db, or a mix of both)

Important notes about deleting many rows at once: Running a single DELETE FROM xxx WHERE height < yyy is not sufficient. It fails in cockroachdb when there are too many rows to be deleted (docs).

What must be done is to re-use the "chunked delete" logic contained in the reset-chain binary, maybe moving it into a shared go pkg.