Deadlock, Goroutines count reaching sky high

[JaskaranSM]

This bug was not present in the v1.47 build I was using, However updated the client today as suggested in https://github.com/anacrolix/torrent/issues/813#issuecomment-1435821697 , After adding around 10 torrents, the runtime started getting goroutine spikes, smells to be a deadlock caused by new changes

Goroutines: 45570
Alloc: 2.4 GB
TotalAlloc: 795.1 GB
HeapAlloc: 2.4 GB
NumGC: 1594

[anacrolix]

Thanks for the report. Are you seeding/uploading a lot? This commit sticks out as a likely candidate: 2cb7121a93c7f534b9d78c98d0f3a3ab7e97043c.

[JaskaranSM]

any update on this? 11 days uptime on application and goroutine count reached below stats:

Alloc: 1.6 GB | TAlloc: 2.9 TB | GC: 14803 | GR: 85654 | TH: 15 | CPU: 25.00% | DL: 2.5 MB | UP: 0 B

[JaskaranSM]

Tested, goroutine leak is not there with 1.48.0 + https://github.com/anacrolix/torrent/commit/e8971ea0f1bfb26e2e05eea6e98a547dfb231bb9

[anacrolix]

If you could provide a stack trace that would be amazing! Also if you want to try running with https://github.com/anacrolix/torrent/commit/2cb7121a93c7f534b9d78c98d0f3a3ab7e97043c reverted/reversed out, that will prove it's the issue too.

[anacrolix]

The goroutine profile mentioned here https://pkg.go.dev/runtime/pprof#Profile, and exposed via https://pkg.go.dev/net/http/pprof is an easy way to capture that. I have a helper that I use do it if you aren't familiar with it: https://github.com/anacrolix/envpprof.

[anacrolix]

@JaskaranSM I believe I have a fix, if the above suspect is indeed the issue. A stack trace would have confirmed it. Please try out this commit: 3c8d70297add48611659626d9d8a583e2448e963.

[JaskaranSM]

Tested latest dev branch, When I tried to directly go get with the aforementioned commit hash, it resulted in unknown revision error (GitHub says its not attached to a branch and could be from outside the repo), Then I just switched to dev because it had that commit, Here's a quick test's results: The client seem to close goroutines fine, The download performance somehow seems to have been improved (yet to look at full commit history of dev) I have also attached the pprof graphs, Envpprof is a nice lib, manually adding pprof handlers in every application seems to be a hassle so great work over there as well. profiles.zip

[anacrolix]

Thank you! I don't know why doing it the envpprof does it isn't the default. I've been doing it this way for nearly 10 years.

[JaskaranSM]

Client panicked for some reason on a torrent. logs: here Magnet: here

[BriungRi]

Can confirm I am observing this issue when running Erigon v2.43.0 which seems to be depending on v1.48.1-0.20230219022425-e8971ea0f1bf

Here's what my goroutine pprof looks like:

Attaching the pprof file as well (zipped to be a supported file format for github) goroutine.pprof.zip

[anacrolix]

@BriungRi please run after go get github.com/anacrolix/torrent@dev. This should be fixed.

[anacrolix]

Client panicked for some reason on a torrent. logs: here Magnet: here

@JaskaranSM Thank you very much!

[anacrolix]

@JaskaranSM are you using any non-standard transports? The panic you show should only occur with WebRTC or webseeding perhaps. I'll have a fix soon regardless.

[anacrolix]

I will tentatively close, this is fixed in dev, and will be in v1.51.0. Let me know if it's not fixed in either of those places!

[AskAlexSharov]

@anacrolix hi. is it possible to backport this fix to v1.48.1 version? because we keep support of go1.19 and version v1.49.0 of torrent lib require go1.20 (so, we can't upgrade for now).

v1.48.1 seems doesn't have netip-addrport.go file

[anacrolix]

I just recently restored compatibility with go 1.19 in v1.51.3 (I also needed it for another project I'm working on). Could you try that?

[AskAlexSharov]

it works, thank you