search

codenotary / immudb

immudb - immutable database based on zero trust, SQL/Key-Value/Document model, tamperproof, data change history
https://immudb.io
Other
8.62k stars 343 forks source link

Index cleanup blocking immudb for several minutes #652

Closed SimoneLazzaris closed 3 years ago

SimoneLazzaris commented 3 years ago

What happened While inserting big volumes of data (1000s of transactions), I've launched an index cleanup. While the cleanup is execution, data ingestion blocks. Access to previous transaction (get, gettx) is also blocking. This can take several (3 or 4) minutes.

What you expected to happen I expected to be able to continue data ingestion and/or perform queries during the reindex or that the block time would be limited to few seconds.

How to reproduce it (as minimally and precisely as possible) Insert data in close loop using execall. Wait until you have several transaction to be cleaned up. Perform an index cleanup from another client. See that data ingestion halts until reindex termination

SimoneLazzaris commented 3 years ago

I will try and provide an executable doing exactly what needed to trigger the issue asap.

mabmayer commented 3 years ago

During testing Metrics and Logs I also observed index cleanup to last for several minutes on a approx. 5 G dataset.

SimoneLazzaris commented 3 years ago

Indexing now blocks just occasionally and for a few seconds. Insertion while indexing is possible. As far as I'm concerned, this issue is solved.