search

rs-ipfs / rust-ipfs

The InterPlanetary File System (IPFS), implemented in Rust.
Apache License 2.0
1.27k stars 165 forks source link

Providing a block takes extremely long amount of time #418

Open semtexzv opened 4 years ago

semtexzv commented 4 years ago

Describe the bug Providing a block to IPFS takes upwards of 4-5 minutes

To Reproduce

let segment_cid = ipfs.put_block(block).await?;
ipfs.provide(segment_cid.clone()).await.unwrap();

Expected behavior Providing a block should be in the realm of tens of seconds

Environment Linux, nightly

Additional context I'm not sure what's the expected performance, but for a service I'm working on, this is simply unbearable. I have 100MB file, and at this rate, just the simple advertisment to the network that my node provides it will take more than an hour.

I'm sure this is not just code problem, but rather problem of bloated DHT, and slow queries within it. But I'd like to raise this issue, and possibly offer my help for working on it.

koivunej commented 4 years ago

Thanks for testing this side out as well!

Providing [...]

The current DHT (libp2p-kad) integration we have has some interop tests and the ipfs-http exposed API is passing some tests at the moment, but the feature as a whole is very much in early steps. Relatedly I can't remember seeing/finding a spec about the the go-ipfs 0.5 split dht and https://github.com/libp2p/specs/pull/108 is still WIP. While implementing the support multiple DHT's wouldn't probably require more than integration code, I am quite sketchy on how/where does the go-ipfs determine to switch the mode, is it just a single dialed in peer through an public IP(v4) and how is that "dial in" "bootstrapped" (lacking a better word for it) or is there just a high probability for it to happen following after a succesful bootstrap?

The bottom line is at the moment we only support only one DHT, which is more catered towards just finding content (see examples/fetch_and_get.rs) or the "dht client" mode where as I understand one doesn't bootstrap but tries to search for content.

Providing a block [...] takes upwards of 4-5 minutes

I'd have to do more digging to better understand why this is taking so long, but at the moment I have only guesses.

I have 100MB file

Supporting providing blocks and such files should very much be on the agenda in the future.