search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
15.84k stars 2.97k forks source link

Sharded directory fetching is unusably slow #4908

Closed ajbouh closed 5 years ago

ajbouh commented 6 years ago

Version information:

0.4.15-dev

Type:

Bug/performance issue

Description:

More context is available over in https://github.com/tesserai/iptf/issues/2

I'm trying to get reasonable performance for just listing the names of entries in a sharded directory that's not yet cached locally. This operation takes hours right now. With @Stebalien's help I've been able to determine that it's only requesting one hash at a time (as indicated by ipfs bitswap wantlist.

Seems like IPFS should be requesting more than one block at a time in this scenario. Creating a separate issue to track this specific performance issue separately from others.

child of #5487

ajbouh commented 5 years ago

@b5 ^

Stebalien commented 5 years ago

This is probably as fast as it's going to get for the foreseeable future.

ajbouh commented 5 years ago

Hi @stebalien, do you mind quantifying how fast things are now?