observablehq / stdlib

The Observable standard library.
https://observablehq.com/@observablehq/standard-library
ISC License
957 stars 83 forks source link

accumulateQuery is slow for fast database clients #318

Closed mbostock closed 1 year ago

mbostock commented 1 year ago

It yields each batch separately:

https://github.com/observablehq/stdlib/blob/6058924f39cb437cf627e5621d493846ebcf6ec7/src/table.js#L224

This adds a minimum of ~17ms between batches. But if the database client is fast, we should coalesce batches into a single yield. We should add a duration test here to determine whether or not to yield.