search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.14k stars 3.02k forks source link

go1.23 timers cause issues with libp2p connmgr and more #10501

Closed lidel closed 3 weeks ago

lidel commented 2 months ago

Version

0.30.0-rc2

Config

default `ipfs init` + custom ports to run A/B test

Description

Bug

0.30.0-rc2 switched to go1.23, and we experience an increased number of connections when running go-libp2p + go-libp2p-kad-dht server, confirmed both by me, and anecdotal evidence from discussion forums.

The problem seem to go away if running with forced go1.22 timer behavior GODEBUG=asynctimerchan=1 (Ref. https://go.dev/wiki/Go123Timer)

How to reproduce

  1. Run two instances of 0.30.0-rc2 with a public IP or UPnP, so they act as DHT servers and get a lot of connections from random peers. Run them with different env variables:
    • GODEBUG=asynctimerchan=0 to force Go 1.23 timers
    • GODEBUG=asynctimerchan=1 to force Go 1.22 timers
  2. After a few hours, the one that runs with GODEBUG=asynctimerchan=0 will have significantly more connections (e.g. 6k vs 200)
  3. Stopping with ctl-C does not work reliably with GODEBUG=asynctimerchan=0, it sometimes takes a second time to correctly stop the daemon.

Next steps

Short term, I will revert https://github.com/ipfs/kubo/pull/10486 and switch back to 1.22 for 0.30.0 release and keep it until go-libp2p and go-libp2p-kad-dht are compatible with 1.23.

Jorropo commented 2 months ago

golang/go#69186 ?

Stebalien commented 2 months ago

I don't think we have any cases where we copy timers.

lidel commented 2 months ago

One of potential causes: https://github.com/quic-go/quic-go/pull/4659 Upstream bug: https://github.com/golang/go/issues/69312 / https://github.com/golang/go/issues/69333

We will wait for upstream go to fix it in new patch release.