anacrolix / torrent

Full-featured BitTorrent client package and utilities
Mozilla Public License 2.0
5.51k stars 622 forks source link

Webseed Client - Close body in same go routine as request #933

Closed mh0lt closed 5 months ago

mh0lt commented 5 months ago

This fix simplifies the go routine profile for http calls.

Instead of routine->channel-routine called one after the other (which is effectively synchronous the code executes in a single go routine.

This is to help elevate this go http related issue - which causes the torrent to panic:

panic: net/http: internal error: connCount underflow

goroutine 1030540 [running]:
net/http.(*Transport).decConnsPerHost(0xc0008dc9a0, {{0x0, 0x0}, {0xc00afd8980, 0x5}, {0xc0274e0360, 0x2f}, 0x0})
    net/http/transport.go:1511 +0x405
net/http.(*Transport).roundTrip(0xc0008dc9a0, 0xc0039eb9e0)
    net/http/transport.go:618 +0x876
net/http.(*Transport).RoundTrip(...)
    net/http/roundtrip.go:17
github.com/ledgerwatch/erigon-lib/downloader.(*requestHandler).RoundTrip(0xc0008dc9a0, 0xc0039eb9e0)
    github.com/ledgerwatch/erigon-lib@v1.0.0/downloader/downloader.go:154 +0x11d
net/http.send(0xc0039eb9e0, {0x18533c0, 0xc0008dc9a0}, {0xc04c4bf501?, 0xc04c4bf788?, 0x0?})
    net/http/client.go:259 +0x5e4
net/http.(*Client).send(0xc000b927b0, 0xc0039eb9e0, {0x523de5?, 0xc0420d0b60?, 0x0?})
    net/http/client.go:180 +0x98
net/http.(*Client).do(0xc000b927b0, 0xc0039eb9e0)
    net/http/client.go:724 +0x8dc
net/http.(*Client).Do(...)
    net/http/client.go:590
github.com/anacrolix/torrent/webseed.(*Client).NewRequest.func1.1.1()
    github.com/anacrolix/torrent@v1.54.1/webseed/client.go:97 +0x27
created by github.com/anacrolix/torrent/webseed.(*Client).NewRequest.func1.1 in goroutine 1030501
    github.com/anacrolix/torrent@v1.54.1/webseed/client.go:96 +0x90
exit status 2

and

panic: net/http: internal error: connCount underflow

goroutine 63612 [running]:
net/http.(*Transport).decConnsPerHost(0xc000f894a0, {{0x0, 0x0}, {0xc02e36ef80, 0x5}, {0xc02b114ed0, 0x2f}, 0x0})
    net/http/transport.go:1511 +0x405
net/http.(*persistConn).closeLocked(0xc032db2900, {0x1852e20, 0x21bae10})
    net/http/transport.go:2764 +0xe5
net/http.(*persistConn).close(0xc000f894a0?, {0x1852e20?, 0x21bae10?})
    net/http/transport.go:2754 +0xa8
net/http.(*Transport).putOrCloseIdleConn(0xc000f894a0?, 0xc032db2900)
    net/http/transport.go:913 +0x2d
net/http.(*Transport).dialConnFor(0xc000f894a0, 0xc015572fd0)
    net/http/transport.go:1491 +0x11d
created by net/http.(*Transport).queueForDial in goroutine 57934
    net/http/transport.go:1461 +0x1cd
exit status 2
AskAlexSharov commented 5 months ago

@anacrolix hi. Do you need my help with reviewing this PR?

anacrolix commented 5 months ago

@AskAlexSharov that would be appreciated!

AskAlexSharov commented 5 months ago

@anacrolix FYI: I created branch release/v1.54 to cherry-pick this PR there and maybe make v1.54.2 tag on that branch. Will take a look maybe need cherry-pick something else.
Diff: https://github.com/anacrolix/torrent/compare/v1.54.1...release/v1.54?expand=1

anacrolix commented 5 months ago

Ok sounds good.