search

netsec-ethz / fpki

4 stars 1 forks source link

ingest tool panic #32

Closed cyrill-k closed 1 year ago

cyrill-k commented 1 year ago

When trying to insert roughly 2 millions certificate (filtered Alexa top 100K), after inserting a few batches, the following error appears. This is done using the ingest tool on this branch: https://github.com/netsec-ethz/fpki/tree/juagargi/ingest_tool

panic: UpdateDomainEntries | doUpdatePairs | wholeBatch | updateFcn | Exec | Error 1213: Deadlock found when trying to get lock; try restarting transaction

With the following backtrace:

goroutine 6156539 [running]:
main.(*BatchProcessor).processBatch(0xc0001ba050, 0xc9813606e0?)
        /home/cyrill/github/fpki/cmd/ingest/batch.go:168 +0x44c
main.(*BatchProcessor).wrapBatch(0xc0001ba050, 0xc9813606e0)
        /home/cyrill/github/fpki/cmd/ingest/batch.go:121 +0x113
created by main.(*BatchProcessor).start.func1
        /home/cyrill/github/fpki/cmd/ingest/batch.go:80 +0x3a
juagargi commented 1 year ago

This update functions are using the old design (well, currently, the only design we have). If deadlocks appear, the quickest way to solve the issue would be to retry the operation on one side.

I would like to have a way to reproduce the issue, to quickly assess if this would fix it or not. Can you give me access to your scenario?

juagargi commented 1 year ago

Closed by #30