search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.05k stars 3.01k forks source link

`ipfs ls` performance #6946

Closed RubenKelevra closed 4 years ago

RubenKelevra commented 4 years ago

Version information:

go-ipfs version: 0.4.23-6ce9a355f Repo version: 7 System version: amd64/linux Golang version: go1.13.7

Description:

I have a two-node setup. Node A got a folder in it's MFS, node B got the cid of this folder.

The folder has more than 10000 files.

HAMT-Sharding is on both sides deactivated.

I can pin the CID very fast - nonrecursive: ipfs pin add -r=0 <cid> returns within 2 seconds.

On the Web-UI I can inspect the CID very fast: Screenshot_20200303_220312

But a simple ipfs ls <CID> takes more than 5 minutes to complete on Node B the first time.

Node A is a server in a datacenter with 4 cores/16 GB Memory, fast RAID10 storage and a load of 0.30 while testing.

The debug output of the DHT shows on node B that node A was discovered an hour ahead of my testing:

DEBUG        dht: PEERS CLOSER -- worker for: Qme3RQdfaDjU5RgUEVfKHrYfoZWQSvq6hf6LdT5Zq4kbSd added QmWsTDap1zmaVLRaUFKBmo25ST6MGtjZQBAT2u72wz4Qma ([/ip6/::1/tcp/4001 /ip6/2a03:4000:34:5c2::f3/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip4/194.59.205.143/tcp/4001]) query.go:309
Stebalien commented 4 years ago

The issue is that ipfs ls fetches the first block of every file in the directory, to determine the type. We've made some improvements but really fixing this involves improving the underlying data structures to embed information like the file type inside the directory itself (one of the goals of unixfs 2.0).

For now, you can use ipfs ls --resolve-type=false --size=false -s.

RubenKelevra commented 4 years ago

Hey @Stebalien this sounds to me like ipfs is currently spending most of the time doing the roundtrips for each request for the first block of every item in the directory.

How about sending a request from Node B to Node A that we need all first blocks of the linked items of a given CID? We don't even need to list the CIDs, we just ask for a recursive fetch with a block size of 1.

Stebalien commented 4 years ago

We currently fetch a bunch in parallel, but not enough.

We're also working on a new protocol, graphsync, that allows us to ask for arbitrary "selectors" over graphs (Filecoin is using graphsync as it's primary data exchange). The new protocol "works" but it can't quite replace bitswap yet because we're missing the smart logic around splitting requests between peers.

RubenKelevra commented 4 years ago

I wasn't really thinking about doing a fetch in parallel, but add a specific requests which just asks for all first blocks of all links of a given CID.

The node A in this example could respond with all first blocks as a continuous stream, just adding a simple bitmap upfront to specify which blocks Node A cannot provide.

This way there's a lot less management needed if Node A got all blocks anyway which is probably pretty likely anyway. If the bitmap is pretty empty we could try a different provider, if we got one for a second chance and then fallback to single requests in parallel.

If we want to support multiple providers for this request we could ask for every n-th first block with a shift variable, this way we could ask for example for every 5-th first block with shift=0 while asking the DHT for a different providers for the second, third, forth and fifths first block.

If there are multiple providers we can query the blocks from them in parallel with just one request needed.

Stebalien commented 4 years ago

I wasn't really thinking about doing a fetch in parallel

I mean that we could be better parallelizing our requests with bitswap as-is.

but add a specific requests which just asks for all first blocks of all links of a given CID.

Please take a look at graphsync.

RubenKelevra commented 4 years ago

but add a specific requests which just asks for all first blocks of all links of a given CID.

Please take a look at graphsync.

Thanks, a deep look at bitswap and the recent changes, just out of curiosity was already on my todo list. I'll take a look at this as well :)

hsanjuan commented 4 years ago

I don't see specific actionables here (other than waiting for matureness and integration of graphsync, ipld). And there is a workaround to speed things up a little bit. Can we close this?

RubenKelevra commented 4 years ago

I don't see specific actionables here (other than waiting for matureness and integration of graphsync, ipld). And there is a workaround to speed things up a little bit. Can we close this?

Yes. Thanks for the heads-up, I forgot to close it.