fix!: replace dag walkers with generic CID extraction from blocks

achingbrain commented 7 months ago

Replace the codec-specific .dagWalkers property with a generic dag walker internally that uses the Block interface from the multicodecs module.

Removes the .dagWalkers property from the Helia interface
Adds getCodec and getHasher to retrieve codecs and hashers by code
Adds loadCodec and loadHasher options to allow sync or async loading of extra codecs/hashes in addition to staticlly configured ones in the codecs/hashers keys

BREAKING CHANGE: the .dagWalkers property has been removed

Change checklist

[x] I have performed a self-review of my own code
[x] I have made corresponding changes to the documentation if necessary (this includes comments as well)
[x] I have added tests that prove my fix is effective or that my feature works

achingbrain commented 7 months ago

I think the pinning benchmarks should be re-run with this PR before merging. The multiformats/block .links() function works by deserializing a block into an object, then recursively walking every property and yielding any value that can be turned into a CID.

This could potentially be more expensive than deserializing a block, collecting CIDs during deserialization and then yielding them all as the DAGWalkers do.

SgtPooki commented 2 months ago

we need to run benchmarks to ensure no perf regression

achingbrain commented 2 weeks ago

Perf seems fine - the benchmark added by https://github.com/ipfs/helia/pull/633 shows comparable performance between this branch and main.

ipfs / helia

fix!: replace dag walkers with generic CID extraction from blocks #447

Change checklist