search

yevhen / Streamstone

Event store for Azure Table Storage
Other
395 stars 64 forks source link

Catch up with stream operation requires full table scan #63

Closed iamim closed 3 years ago

iamim commented 3 years ago

Stream.ReadOperation method that is used to catch up with the stream uses OR to fetch HEAD and latest entries. It turns out that Azure Table queries containing OR result in full table scan. As a result, read latency grows linearally with increasing number of entries. I can't provide exact numbers (Azure logs are not avaialble anymore) but it was roughly equal to 1s for 100k rows. This is true even if the event hasn't changed.

Would it be possible to modify this opeartion to not include OR clause? Potentially split this operation into 2 requests (head then slice) if version is greater than x (10,000?). In this case the second request could be skipped if returned head is equal to the in-memory stream.

yevhen commented 3 years ago

This is already fixed in recent version https://github.com/yevhen/Streamstone/pull/61

iamim commented 3 years ago

My bad, haven't noticed it. Had to work around this issue about half a year ago and only now got myself to post it here.

Great library! Thank you!