search

anacrolix / torrent

Full-featured BitTorrent client package and utilities
Mozilla Public License 2.0
5.58k stars 630 forks source link

Webseed in https not working #966

Closed gatisahu closed 2 months ago

gatisahu commented 3 months ago

HTTP webseed is working fine. HTTPs webseed sending request to webseed server and receiving the chunk but it is not progressing .

we are using response.addHeader("Accept-Ranges", "bytes");

After debugging it seems some peice the pieceHashed coming false https://github.com/anacrolix/torrent/blob/master/torrent.go#L2379 While checking I found comparing sha1 hash it is not matching . Not sure how https playing a role here

tls config

tlsConfig := &tls.Config{ RootCAs: caCertPool, InsecureSkipVerify: false, // Verify server's certificate MinVersion: tls.VersionTLS12, }

cfg.WebTransport = &http.Transport{ TLSClientConfig: tlsConfig, }

full status

Torrents: 1

p0.51297892-el8.parcel 0.017206% of 13997539212 bytes (14 GB) Infohash: ea34737fb4f35c2f5d5161a911f45befa1cb1d3a Metadata length: 33495 Piece length: 8388608 (512 chunks) Num Pieces: 1669 (0 completed) Piece States: 1. 1.P 5. 1.P 1661. Piece availability frequency: 1: 1669 Reader Pieces: Enabled trackers: URL Extra DHT Announces: 0 (torrent.TorrentStats) { ConnStats: (torrent.ConnStats) { BytesWritten: (torrent.Count) 1545, BytesWrittenData: (torrent.Count) 0, BytesRead: (torrent.Count) 1545, BytesReadData: (torrent.Count) 0, BytesReadUsefulData: (torrent.Count) 0, BytesReadUsefulIntendedData: (torrent.Count) 0, ChunksWritten: (torrent.Count) 0, ChunksRead: (torrent.Count) 2672275, ChunksReadUseful: (torrent.Count) 2672275, ChunksReadWasted: (torrent.Count) 0, MetadataChunksRead: (torrent.Count) 0, PiecesDirtiedGood: (torrent.Count) 0, PiecesDirtiedBad: (torrent.Count) 0 }, TotalPeers: (int) 0, PendingPeers: (int) 0, ActivePeers: (int) 0, ConnectedSeeders: (int) 0, HalfOpenPeers: (int) 0, PiecesComplete: (int) 0 } webseeds:

gatisahu commented 2 months ago

This is not specific to https issue. When I am using v1.56.0 I am seeing below error

banning webseed peer for "https://######/parcel/download/some.parcel" for being sole dirtier of piece 6 after failed piece check [ github.com/anacrolix/torrent torrent.go:2458 ]

I have tried to run master I don't see the same issue. Still I see when there is a gap the webseed don't download. just for testing I have added ws.peer.updateRequests("onSetInfo") in addWebSeed method and it is working . I think when request order happen it call webseed and if webseed is added after this then webseed don't do anything .

anacrolix commented 2 months ago

See https://github.com/anacrolix/torrent/issues/964#issuecomment-2295308596.

gatisahu commented 2 months ago

The code is wait on ws.requesterCond.Wait() .Even if signal to wake up ws.peer.requestState.Requests will be null ,so it will not go inside critical section. https://github.com/anacrolix/torrent/blob/master/webseed-peer.go#L97

On torrent during addwebseed adding ws.peer.updateRequests("onSetInfo") is creating request and working fine . This will cause any regression ?

anacrolix commented 2 months ago

This should be fixed in https://github.com/anacrolix/torrent/commit/3b0b61fa963a67b1b02b0e036f2f7b22d649eecf.

gatisahu commented 2 months ago

Thanks for fixing . Now webseed is working fine in delayed case.Now it is breaking the normal use case where I have tested with 5 nodes and adding webseed after every 5 secs. Without the code changes it is working fine.

We add add peer every 90 secs.

With the changes from 5 nodes 3 nodes only download from webseed and not exchange message through peer .Two did not start after restart it start again . I will test little bigger cluster and update .

seeing message in log 2024-08-27T08:31:09.516Z debug log/logger.go:157 error receiving handshakes on *torrent.PeerConn 0xc00061ea88 [flags=I,U,v1 id="\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00", exts=0000000000000000 (), v=%!q()]: EOF [ github.com/anacrolix/torrent client.go:1066 ]

full status Peer ID: "-GT0003-rhyL\xdau\xdc\x1ef\xe7'-" Extension bits: 0000000000100005 (ltep, fast, dht) Announce key: 66e7272d Banned IPs: 0 (torrent.ClientStats) { ConnStats: (torrent.ConnStats) { BytesWritten: (torrent.Count) 123616, BytesWrittenData: (torrent.Count) 0, BytesRead: (torrent.Count) 8405544849, BytesReadData: (torrent.Count) 8405385216, BytesReadUsefulData: (torrent.Count) 8405385216, BytesReadUsefulIntendedData: (torrent.Count) 8405385216, ChunksWritten: (torrent.Count) 0, ChunksRead: (torrent.Count) 1002, ChunksReadUseful: (torrent.Count) 1002, ChunksReadWasted: (torrent.Count) 0, MetadataChunksRead: (torrent.Count) 0, PiecesDirtiedGood: (torrent.Count) 1002, PiecesDirtiedBad: (torrent.Count) 0 }, ActiveHalfOpenAttempts: (int) 0, NumPeersUndialableWithoutHolepunch: (int) 0, NumPeersUndialableWithoutHolepunchDialedAfterHolepunchConnect: (int) 0, NumPeersDialableOnlyAfterHolepunch: (int) 0, NumPeersDialedSuccessfullyAfterHolepunchConnect: (int) 0, NumPeersProbablyOnlyConnectedDueToHolepunch: (int) 0 }

Torrents: 1

CDH-7.3.1-1.cdh7.3.1.p0.56439301-el8.parcel 59.563233% of 14111700715 bytes (14 GB) Infohash: 58f1a52e8c0315a8a2cce6ea8b755a1fcb1a214c Metadata length: 33773 Piece length: 8388608 (0.0625 chunks) Num Pieces: 1683 (1002 completed) Piece States: 918C 30. 84C 651. Piece availability frequency: 5: 1683 Reader Pieces: Enabled trackers: URL Extra DHT Announces: 0 (torrent.TorrentStats) { ConnStats: (torrent.ConnStats) { BytesWritten: (torrent.Count) 123616, BytesWrittenData: (torrent.Count) 0, BytesRead: (torrent.Count) 8405544849, BytesReadData: (torrent.Count) 8405385216, BytesReadUsefulData: (torrent.Count) 8405385216, BytesReadUsefulIntendedData: (torrent.Count) 8405385216, ChunksWritten: (torrent.Count) 0, ChunksRead: (torrent.Count) 1002, ChunksReadUseful: (torrent.Count) 1002, ChunksReadWasted: (torrent.Count) 0, MetadataChunksRead: (torrent.Count) 0, PiecesDirtiedGood: (torrent.Count) 1002, PiecesDirtiedBad: (torrent.Count) 0 }, TotalPeers: (int) 4, PendingPeers: (int) 0, ActivePeers: (int) 4, ConnectedSeeders: (int) 4, HalfOpenPeers: (int) 0, PiecesComplete: (int) 1002 } webseeds:

anacrolix commented 2 months ago

It looks like connectivity issues because you're downloading so fast!

gatisahu commented 2 months ago

Webseed I have increased the chunk size 16 kb to 1 mb . You are suspecting it is choking network bandwidth of host downloading from webseed ?

anacrolix commented 2 months ago

Yes

gatisahu commented 2 months ago

Thanks it is working fine .Now default chunk size is 16 kb which is very low. Is there a specification or recommended chunk size for webseed

anacrolix commented 2 months ago

It's recommended to leave the chunk size at the default of 16 KiB and only increase if you are seeing significant overhead or running your own swarm.

gatisahu commented 2 months ago

After decreasing it is working fine .Now I am testing a larger cluster in aws where there is multiple AZ so Our logic is every rac will have a rac leader and download from webseed .So is there way I can only increase the chunk size of webseed only do at least download 1% of file in one http request else the webseed server will rate limit it .If you can provide some code pointer and your thought I can try the implementation and verify in aws .

anacrolix commented 2 months ago

It's not exposed as a config option, but you would want to modify this

https://github.com/anacrolix/torrent/blob/dcfb61fe6587fdc89359f8603c157e94ad8f4245/torrent.go#L2860

And increase chunk size 😄