Repro

[anacrolix]

Thanks for the input. If I'm reading your log however, it works fine for you? I need it to demonstrate failure so I can find what is wrong.

[abhishek-das-gupta]

Thanks for the input. If I'm reading your log however, it works fine for you? I need it to demonstrate failure so I can find what is wrong.

@anacrolix Thank you very much for providing this test case! But I'm not sure what do you mean by "failure". There is no failure per se. The #953 issue is related to slow performance not failure.

If you look at tests/peers-bootstrapping/Client-Torrent.log in the PR, you can see only Client0 making progress while other Client[1..5] are all stuck at 0% progress until the Client0 reaches 100%. This is what I described in #953. I added the full-status of all the node in the comment

Although I'm not sure if the changes that I added in main.go in commit1 are correct or not. I guess since I made the initalTorrent completion as a blocking call due to which other peers are not progressing?

I've added another commit right now, which seems to be as closes to do what we are doing in our code.

source dir <---- is where the node0 downloads "file" binary.
client0 dir<----- is the cfg.DefaultStorage directory for the node0 or client0
client1--------------
.                    |
.                    | -> all the clients add the WebSeed called file://source/file (In my code, it would be a HTTP url )
.                    |
client5 -------------

When I make these changes, the test main.go seems to stuck. I'm unable to debug this? Can you please provide insights what am I doing wrong? Thanks

[anacrolix]

Thanks! I'll take a look

[anacrolix]

Well I spotted a masked bug in the webseed code. However that's not the issue.

The things I fixed:

You can't use file:// for a webseed URL. Spin up a HTTP server if you want to do that (do it in Go, in the repro, easy).

You can't not seed the actual completed data if webseeding isn't available.

You can't copy over the storage files while they're in use. You need to do it before the torrent's storage is active.

At least one client needs to have the actual full metainfo. You can't get this over webseeding, you need to use the metainfo.

[anacrolix]

Could you make the Go code reproduce this? I can't see anything wrong with your code at a glance, but it's not straightforward.

[abhishek-das-gupta]

Hi @anacrolix , Is there a client config which lets me change the download request size when downloading from web seed? Like suppose I want to request 1 MiB at a time from webseed?

[anacrolix]

The webseeding code mirrors the regular code in the size of chunks it attempts. Look for "chunksize" in the client or torrent config. I know a few users set it very high (1MiB or so). There's no way to set it just for webseeding as webseeds and regular conns share the same request algorithm.

[abhishek-das-gupta]

Thanks, @anacrolix, for your test case! Using your testcase, once I set the optimal piece length, the torrent completed quickly. However, I have one question: suppose there is no seeder to begin with (another use case). My fallback is to contact the webseed to download the file.

All the peers within the swarm contact the single webseed, but I was hoping they would also communicate among each other to distribute the pieces. However, it seems like this is not happening, as I don't see any activity under requested pieces:. Please see the following status of peer #4. You can see that it is not requesting any pieces from its peers, resulting in a sequential HTTP download.

<some-parcel>.parcel
16.575733% of 4048621206 bytes (4.0 GB)
Infohash: 075b6c84d4bb6e83046f6b14c77b075de307e318
Metadata length: 38741
Piece length: 2097152 (128 chunks)
Num Pieces: 1931 (320 completed)
Piece States: 2C 1. 2C 1. 8C 1. 1C 1. 4C 1. 4C 1. 5C 1. 1C 1. 18C 1. 3C 1. 5C 1. 5C 1. 6C 1. 7C 1. 1C 1. 6C 1. 31C 1. 18C 1. 10C 1. 8C 1. 3C 1. 4C 1. 9C 1. 23C 1. 51C 1. 15C 1. 12C 1. 22C 1. 15C 1. 1C 1. 20C 1581.
Piece availability frequency: 0: 1611, 4: 320
Reader Pieces:
Enabled trackers:
    URL  Extra
DHT Announces: 0
(torrent.TorrentStats) {
 ConnStats: (torrent.ConnStats) {
  BytesWritten: (torrent.Count) 288237498,
  BytesWrittenData: (torrent.Count) 287277056,
  BytesRead: (torrent.Count) 672103818,
  BytesReadData: (torrent.Count) 671219712,
  BytesReadUsefulData: (torrent.Count) 671088640,
  BytesReadUsefulIntendedData: (torrent.Count) 671088640,
  ChunksWritten: (torrent.Count) 17534,
  ChunksRead: (torrent.Count) 40968,
  ChunksReadUseful: (torrent.Count) 40960,
  ChunksReadWasted: (torrent.Count) 8,
  MetadataChunksRead: (torrent.Count) 0,
  PiecesDirtiedGood: (torrent.Count) 320,
  PiecesDirtiedBad: (torrent.Count) 0
 },
 TotalPeers: (int) 4,
 PendingPeers: (int) 0,
 ActivePeers: (int) 4,
 ConnectedSeeders: (int) 0,
 HalfOpenPeers: (int) 0,
 PiecesComplete: (int) 320
}
webseeds:
4 peer conns:
- 10.65.55.225:7191-10.65.54.157:52422
  peer id: "-GT0003-[\t\x1b\xb6\x00\xeaa\xaf\x97h\x9a\xec"
  extensions: 0000000000100005 (ltep, fast, dht)
  ltep extensions: map[ut_holepunch:2 ut_metadata:1 ut_pex:3]
  pex: 4 conns, 0 unsent events
  bep40-prio: f3cfa364
  last msg: 0.11s ago, connected: 84.55s ago, last helpful: 0.11s ago, itime: 1m24.544555365s, etime: 4.811743861s
  320/1931 completed, 0 pieces touched, good chunks: 40561/40561:0 reqq: 0+0/(1/1024):0/1024, flags: i:I,e,v1:, dr: 134873.3 KiB/s
  requested pieces:
- 0.0.0.0:7191-10.65.51.253:7191
  peer id: "-GT0003-_-6\x8b\xac^\x95\x83\x06x\xdbA"
  extensions: 0000000000100005 (ltep, fast, dht)
  ltep extensions: map[ut_holepunch:2 ut_metadata:1 ut_pex:3]
  pex: 4 conns, 0 unsent events
  bep40-prio: ef9c4773
  last msg: 0.10s ago, connected: 84.60s ago, last helpful: 20.98s ago, itime: 578.115117ms, etime: 486.698749ms
  320/1931 completed, 0 pieces touched, good chunks: 23/31:272 reqq: 0+0/(1/1024):0/1024, flags: :I,U,e,v1:i, dr: 756.1 KiB/s
  requested pieces:
- 10.65.55.225:7191-10.65.49.106:54548
  peer id: "-GT0003-\x93D6\x8cN\x99$%E\x04\xaf\x1d"
  extensions: 0000000000100005 (ltep, fast, dht)
  ltep extensions: map[ut_holepunch:2 ut_metadata:1 ut_pex:3]
  pex: 4 conns, 0 unsent events
  bep40-prio: ca339400
  last msg: 0.10s ago, connected: 84.60s ago, last helpful: 13.01s ago, itime: 1m24.434434388s, etime: 65.271288ms
  320/1931 completed, 0 pieces touched, good chunks: 254/254:1335 reqq: 0+0/(1/1024):0/1024, flags: i:I,e,v1:i, dr: 62263.2 KiB/s
  requested pieces:
- 10.65.55.225:7191-10.65.50.36:46832
  peer id: "-GT0003-\x8a\xfd\xf8\xaf\x95\xbce\xf1h=\x82\xd2"
  extensions: 0000000000100005 (ltep, fast, dht)
  ltep extensions: map[ut_holepunch:2 ut_metadata:1 ut_pex:3]
  pex: 4 conns, 0 unsent events
  bep40-prio: 5f7086d4
  last msg: 0.07s ago, connected: 84.59s ago, last helpful: 0.09s ago, itime: 1m9.563634782s, etime: 9.983955ms
  320/1931 completed, 0 pieces touched, good chunks: 122/122:15927 reqq: 0+0/(1/1024):0/1024, flags: i:I,e,v1:i, dr: 195513.7 KiB/s
  requested pieces:

When I switch to Libtorrent, I see that peers simultaneously download from the webseed and distribute pieces among themselves.

Is there a way to achieve this in your torrent library?

[anacrolix]

It looks like your peers are downloading in lockstep. If you look at their "x/y completed", they're all at the same value. They're sharing faster than it takes to request more pieces (I'm guessing they're on the same local network or host). The requested pieces field shows active requests. At the time the status was generated there are no active requests. If you look at "good chunks" that shows how many requests were satisfied from the peer since they last connected (and how many they uploaded to that same peer).

[anacrolix]

Thanks for working through this, and the contribution!