Performance indexing "big directories"

ipfs / kubo

An IPFS implementation in Go

https://docs.ipfs.tech/how-to/command-line-quick-start/

Other

16.14k stars 3.02k forks source link

Performance indexing "big directories" #7588

Open kallisti5 opened 4 years ago

kallisti5 commented 4 years ago

Version information:

$ ipfs version --all go-ipfs version: 0.6.0 Repo version: 10 System version: amd64/linux Golang version: go1.14.4

Description:

Usage:

Add a folder containing 35GiB of files across ~7,108 files.
Documents only exist on a single node with a 200 Mbit up/down internet connection
Navigate to folder on a gateway such as gateway.ipfs.io (success)
Enter the folder (index) with 7,108 files. Timeout on gateway.

Does the IPFS gateway need to transfer a large amount of data to get a directory index?

The IPFS daemon has been running about 30Mbit/s for a while now. bandwidth

kallisti5 commented 4 years ago

I connected a remote node (vm at external hosting provider) directly to my local home IPFS node: ipfs swarm connect /ip4/XX.XX.XX.XX/tcp/4001/p2p/QmdDXXX

and ran an ipfs ls...

time ipfs ls /ipfs/QmWApXeXXX/current/packages

It took over 7 minutes to perform an ls:

real 7m17.463s

utkarsh5k commented 4 years ago

I'm willing to take a look at this, but I'm a first time contributor, so if someone could point me to the indexing logic, that'd be super helpful, thanks!

aschmahmann commented 4 years ago

Does the IPFS gateway need to transfer a large amount of data to get a directory index?

It currently needs the to get the first block of each file in the directory to decide if it's a file or a directory.

As for ipfs ls if you set --resolve-type and --size to false it will only need to get the directory data and not reach into the individual files. Also having --stream will give you outputs without having to wait for the full results.