search

ipfs / js-ipfs-unixfs

JavaScript implementation of IPFS' unixfs (a Unix FileSystem representation on top of a MerkleDAG)
Other
86 stars 33 forks source link

feat: add blockReadConcurrency option to exporter #361

Closed achingbrain closed 7 months ago

achingbrain commented 11 months ago

By default we attempt to load all sibilings in a given layer of a DAG at once to allow slow/async loading routines extra time to fetch data before it is needed.

Some blockstores (e.g. CAR files) require the exporter to only request the next sequential CID in a DAG.

Add a blockReadConcurrency option (named similarly to the importer's blockWriteConcurrency option) to control this behaviour.

Fixes #359

achingbrain commented 11 months ago

cc @guanzo

guanzo commented 11 months ago

Does the concurrency option also need to be set in these 2 places?

https://github.com/ipfs/js-ipfs-unixfs/blob/88d73a6f1bb30af84ea90286145ad1b894261520/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/directory.ts#L28 https://github.com/ipfs/js-ipfs-unixfs/blob/88d73a6f1bb30af84ea90286145ad1b894261520/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L65

achingbrain commented 7 months ago

@guanzo yes, good spot - this has been updated and tests for this added.

github-actions[bot] commented 7 months ago

:tada: This PR is included in version 13.4.0 :tada:

The release is available on npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:

github-actions[bot] commented 5 months ago

:tada: This PR is included in version 11.1.4 :tada:

The release is available on npm package (@latest dist-tag)

Your semantic-release bot :package::rocket: