Feat: improve memory usage / allow for in-memory storage of all entries to be optional (WIP)

mistakia commented 4 years ago

⚠️ Note: This PR is a WIP and not ready for review. I'm in the process of cleaning up a fork that I have been using in a proof of concept application. My hope is to minimize the amount of breaking changes further.

Description

Currently, all entries (and all the data in it) are kept in memory. The goal of this PR is to make this optional in order to improve load times and performance on memory-limited environments when interacting with larger/longer log sets.

Breaking Changes

The following getters and methods become async / return promises.

log.values
log.tails
log.tailHashes
log.get()
log.traverse()
log.iterator()
log.toString()
log.toSnapshot()

Benchmarking

Before wip

After wip

Tradeoffs

wip

TODO

[x] update tests
[ ] write new tests
[ ] update benchmarks / benchmarking / document performance tradeoffs
[x] adjust entryIndex to allow caching of all entries by default
[ ] document breaking changes
[x] update method comments
[x] squash / clean up commits
[x] remove forked orbit-db-io

aphelionz commented 4 years ago

Excited about this if it turns out to work! Would you mind though making this a draft PR until it's ready?

aphelionz commented 4 years ago

How are you feeling about this one @mistakia ? :)

mistakia commented 4 years ago

Feeling like I need to get back in the game! I plan on coming back to this in the not so distant future.

aphelionz commented 4 years ago

Ok, I'll leave it open then :+1:

aphelionz commented 3 years ago

@mistakia the not too distant future is upon us :D hope you are happy + healthy in these trying times.

If you want to hop back in I can totally support you. Did you see I finally merged #307 ?

mistakia commented 3 years ago

Quick status update of what I think is left. Merge the following into ipfs-log master:

After these two I can dig deeper in the benchmarks to make improvements (i.e. analyze various LRU libs)

[x] update log.spec tests
[ ] what to do with errors when loading an entry from a multihash
[ ] document breaking changes
[x] update docs

orbitdb-archive / ipfs-log