search

ipfs-rust / ipfs-embed

A small embeddable ipfs implementation
344 stars 47 forks source link

Add read and write batches... #86

Closed rklaehn closed 2 years ago

rklaehn commented 2 years ago

For internals, this has been available for a while via the block store, even though we did not yet take advantage of it.

With this API it is also available for ipfs-embed users, such as actyx.

The concrete use case is to be able to add all blocks of a fast path update in a single transaction, or to be able to do a banyan rewrite fully in memory and only spill over to disk once the memory level is exceeded or the write is done.

Note that there is no overhead by opening transactions. sqlite opens a transaction anyway, and in the block store all ops are implemented with a single statement transaction as well.

EDIT: this is WIP until there are tests, but I want to know what you think of the approach

rklaehn commented 2 years ago

with this I could make some parts of actyx higher performance. more batching in bitswap could have been done earlier, since ipfs-sqlite-block-store offers transactions for a while. The tricky thing there is to batch somehow despite the protocol not being batch based.

rklaehn commented 2 years ago

In principle we could just remove all ops except read_batch and write_batch from ipfs-embed. Not sure if we should keep the individual ops or remove them.

dvc94ch commented 2 years ago

I don't think we should have breaking changes that aren't necessary. Adding new stuff is fine.