mediachain / concat

Mediachain daemons
MIT License
42 stars 13 forks source link

Fix Delete deadlock when connection pooling is disabled #27

Closed vyzo closed 7 years ago

vyzo commented 7 years ago

So in order to not kill concurrent write performance, sqlite needs to be used with conn pooling disabled. Unfortunately this causes Delete to deadlock, as it is using a streaming query to fetch the ids for deletion. On the other hand simply fetching all the ids together in memory can lead to problems for large datasets (too much memory). So in order to have our cake and eat it too (at least most of it), delete operates in batches. The ugly side is that we can now have partial deletes, which are distinguished by both the returned count being >0, and the error being non-nil.

In other changes, the node has stopped being chatty in statement publication.

parkan commented 7 years ago

Unfortunate that we have to do this but impl looks good to me