search

lbryio / reflector.go

MIT License
14 stars 13 forks source link

Insert in tx #51

Closed lyoshenka closed 2 years ago

lyoshenka commented 3 years ago

my version of @shyba's #50. see which you like better (or maybe you'll have an even better idea?)

wrap blob insertion in tx. fixes lbryio/lbry-sdk#3296

The problem is that inserting an sd blob with ~5k blobs takes longer than 30 seconds. So the client times out and retries the request. At that point, reflector is not done inserting so it replies with a smaller number of blobs than it should. The client uploads that many blobs and marks the stream as reflected. The remaining blobs never get uploaded.

Victor says doing the insert inside a transaction should be faster than doing 10k (2 per blob) inserts independently.

He's also doing a client-side fix for this.

shyba commented 3 years ago

client side fix at https://github.com/lbryio/lbry-sdk/pull/3308

if we can make that faster it should also help not having incomplete data on shutdown. I had a similar issue on the SDK some years ago and using a transaction was absurdly faster, but I don't know how much that replicates to MySQL

anbsky commented 3 years ago

I'm in favor of this implementation. Aside from a few nitpicks, I don't have any qualms with it.