search

sourcenetwork / defradb

DefraDB is a Peer-to-Peer Edge Database. It's the core data storage system for the Source Network Ecosystem, built with IPLD, LibP2P, CRDTs, and Semantic open web properties.
399 stars 40 forks source link

Rethink aggregate options (limit etc), grouping, and inline array flow #770

Open AndrewSisley opened 2 years ago

AndrewSisley commented 2 years ago

At the moment limt, filter etc have to be explicitly performed within aggregate nodes. This is due to both inline arrays (no current way to stream their items through nodes, e.g. limit), and because of the way grouping works (single pass).

Have a bit of a think on this and see if we can remove some of this duplication - it will continue to grow as more aggregates get added, and as more options get added to them.

AndrewSisley commented 2 years ago

Having a look at this for 0.3.1, might descope it again if I think it is too much

AndrewSisley commented 2 years ago

I thought about it, is not worth doing this now, especially given that arrays are all read into memory at the moment anyway. Might include additional locally-scoped refactorings with the array-order ticket, but nothing significant for 0.3.1