splitgraph / seafowl

Analytical database for data-driven Web applications 🪶
https://seafowl.io
Apache License 2.0
386 stars 9 forks source link

Best effort chunk coalescing implementation #515

Closed gruuya closed 2 months ago

gruuya commented 2 months ago

This is a naive implementation of https://github.com/splitgraph/seafowl/pull/514 (and therefore less complex), where we don't use synchronization to avoid request de-duplucation, but instead try do a best-effort chunk coalescing, and batch them into a single call.

Compared to current main, this implementation seems to be more stable, and doesn't experience the hanging in TPC-H q18 and q21, since it does not use any synchronization primitives.

As for perf, in the (idealized) no latency case it is on par with the current caching object store. However in more realistic scenarios of modest delays (15ms) it consistently outperforms it: image