@peerbit/share-log: `distribute()` is memory in-efficient - Githubissues

dao-xyz / peerbit

P2P database framework with encryption, sharding and search

https://peerbit.org

Apache License 2.0

188 stars 14 forks source link

@peerbit/share-log: `distribute()` is memory in-efficient #324

Closed marcus-pousette closed 1 week ago

marcus-pousette commented 1 month ago

This line

https://github.com/dao-xyz/peerbit/blob/52f4e300eee9323f81b167f7c585ab53b00a6891/packages/programs/data/shared-log/src/index.ts#L2225

expects all heads to be in memory.

Additionally, this does not scale well once many entries are added.

TODO

Make a RAM conservative solution
Use peer/join leave change sets to reduce the amount of heads we need to iterate over
- Replication domains need to implement some kind of differential scopes/content space range to reduce the search width for data rebalancing

marcus-pousette commented 1 week ago

Solved in #330