Closed SOVLOOKUP closed 1 year ago
I've reproduced the problem - after some 190 clients are spawned. The error seems to vary. With a compiled version it was as SOVLOOKUP reported. Debugging to try and find the source of the error - I've seen "received data 503", "dial backoff". My suspicion is that the error is ultimately coming from go-ipfs - and it's "point of entry" is in p2p/host/basic/basic_host.go - line 736 in the dialPeer method. Having said that I'm unsure whether it's a valid use case? Should a tight loop be able to spawn hundreds/thousands of clients without pause or error?
I took your repro and used stock go-libp2p parts and I can't repro. Maybe you're hitting into your resource limits set by the resource manager?
Does the following also fail for you?
https://gist.github.com/MarcoPolo/ab646ca5e8853a560a94ce7727daacce
@vernonr3 In the real-world application, my device may communicate with hundreds/thousands of peers at the same time. I want to test the upper limit of peers that libp2p can connect to.
After running the example given by @MarcoPolo, I found that libp2p seems to be able to connect to a maximum of about 1000 peers.
If you change the limit of 1024 to 2048, the program will throw a panic: stream reset
error when the number of connections reaches about 1000
limits := rcmgr.PartialLimitConfig{
System: rcmgr.ResourceLimits{
Conns: 2048,
ConnsInbound: 2048,
},
Transient: rcmgr.ResourceLimits{
Conns: 2048,
ConnsInbound: 2048,
},
}
@SOVLOOKUP Can we separate the number of peers to which your device "may" communicate from the speed at which connections are created? Your sample code had a delay of 1/10 second between spawning goroutines. What made you choose this particular value? I'm finding that the time delay between connections being created seems to have an effect on what happens. I'm aware this might well be hardware/OS specific.
Celestia observes a similar issue on the production network with ~1000 peers. On start, our nodes connect to bootstrappers, and some cannot connect due to dial backoff
. We have infinite resource limits and a high connection manager threshold on them, so why that would happen is unclear.
I've done further work on this taking @MarcoPolo' s code as a vital basis. There appears to be a problem with rapidly spawning lots of connections. Not always perhaps (it might depend on OS/hardware etc.)
Two branches in my repo (spawnrate folder) worth looking at:
a) Pattern of missing connections... https://github.com/vernonr3/libp2p_tests/tree/inside_goroutine/spawnrate This adds instrumentation and parameterizes a delay inside a goroutine before creating connections. The "interesting results" (piping stdout to file) can be seen in MissingDelay150.txt, MissingDelay200.txt and MissingDelay250.txt. The "missing connections" are those which approach each multiple of the delay factor. (see commit eb285ca for further details). This doesn't "feel right". Something nasty is happening...
b) How many connections are missed and why https://github.com/vernonr3/libp2p_tests/tree/outside_goroutine/spawnrate Here the delay has been moved outside the goroutine. This stops the bunching that happened in case a - and produced a more even spawn rate.
Counting the number of "start dialing" vs "finished dialing" debug messages in DialingCompare4.txt (these debug messages are from inside p2p/host/basic/basic_host.go - see commit 2f42cfc for more details of what I did ). My belief (based on documentation) is that the ProtocolIdentify (handshake3) isn't completing correctly in the missing cases. Don't know why as yet..
Further work...
See spawnrate2
The output is in the text file output.txt
Added extra logging for a local copy of the golibp2p library Experimented... Findings thus far:
Closing the client (at the end of newClient) gets rid of the problem Setting the IP listen address deliberately to 127.0.0.1 deliberately improves the situation slightly - it can manage 1000 connections but not many more
There large numbers of ErrReset issued from transport/quic/stream.go in the Read and Write Functions - which are seen as "stream reset" on the client end
At a higher level the error appears within the client call to SelectProtoOrFail. The actual error occurs in quic-go/receive_stream.go _) I think as a result of cancelRead....
Since the server is still alive and clients are also alive (even if quiet after their initial ping) I'm not sure why these cancellations are happening?
@vernonr3, relation to SelectProtoOrFail
reminds me of the issue my team had. The root of the problem was buggy code on our side. We didn't handle EvtPeerConnectednessChanged
and it blocked the Identify protocol on SelectProtoOrFail
blocking any other new stream. Double-check your event handling
This issue was a problem on our side, again. go-libp2p works perfectly, as always 😺
Celestia observes a similar issue on the production network with ~1000 peers. On start, our nodes connect to bootstrappers, and some cannot connect due to dial backoff. We have infinite resource limits and a high connection manager threshold on them, so why that would happen is unclear.
@vernonr3 I'm guessing you're hitting resource manager limits for total number of active streams for ping. The resource manager is meant to protect you from getting dos'd but if you want to see how many connections you can spin up to your self as fast as possible try this diff:
diff --git a/spawnrate2/main.go b/spawnrate2/main.go
index e71afb2..33aaa95 100644
--- a/spawnrate2/main.go
+++ b/spawnrate2/main.go
@@ -6,44 +6,33 @@ import (
"fmt"
"sync"
"sync/atomic"
- "time"
logging "github.com/ipfs/go-log/v2"
libp2p "github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p/core/host"
+ "github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
- rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"
"github.com/libp2p/go-libp2p/p2p/net/connmgr"
"github.com/libp2p/go-libp2p/p2p/protocol/ping"
)
func setupLogging() {
- err := logging.SetLogLevel("ping", "debug")
- if err != nil {
- panic(err)
- }
- err = logging.SetLogLevel("upgrader", "debug")
- if err != nil {
- panic(err)
- }
- err = logging.SetLogLevel("quic-transport", "debug")
- if err != nil {
- panic(err)
- }
- err = logging.SetLogLevel("webtransport", "debug")
+ var err error
+ err = logging.SetLogLevel("quic-transport", "warn")
if err != nil {
panic(err)
}
-
}
func controlclients(s host.Host, clients int) {
+ sem := make(chan struct{}, 8)
n := atomic.Int32{}
wg := sync.WaitGroup{}
for i := 0; i < clients; i++ {
wg.Add(1)
go func(i int) {
- time.Sleep(time.Duration(i%100) * 100 * time.Millisecond)
+ sem <- struct{}{}
+ defer func() { <-sem }()
defer wg.Done()
err := newClient(peer.AddrInfo{
ID: s.ID(),
@@ -68,30 +57,7 @@ func main() {
clients := flag.Int("clients", 1000, "Set number of clients")
flag.Parse()
fmt.Printf("Running with setip=%v and number of clients %d\n", *setip, *clients)
- limits := rcmgr.PartialLimitConfig{
- System: rcmgr.ResourceLimits{
- Streams: rcmgr.LimitVal((*clients) * 2),
- StreamsInbound: rcmgr.LimitVal((*clients) * 2),
- StreamsOutbound: rcmgr.LimitVal((*clients) * 2),
- Conns: rcmgr.LimitVal((*clients) * 2),
- ConnsInbound: rcmgr.LimitVal((*clients) * 2),
- ConnsOutbound: rcmgr.LimitVal((*clients) * 2),
- },
- Transient: rcmgr.ResourceLimits{
- Streams: rcmgr.LimitVal((*clients) * 2),
- StreamsInbound: rcmgr.LimitVal((*clients) * 2),
- StreamsOutbound: rcmgr.LimitVal((*clients) * 2),
- Conns: rcmgr.LimitVal((*clients) * 2),
- ConnsInbound: rcmgr.LimitVal((*clients) * 2),
- ConnsOutbound: rcmgr.LimitVal((*clients) * 2),
- },
- }
-
- limiter := rcmgr.NewFixedLimiter(limits.Build(rcmgr.DefaultLimits.AutoScale()))
- rmgr, err := rcmgr.NewResourceManager(limiter)
- if err != nil {
- panic(err)
- }
+ rmgr := &network.NullResourceManager{}
cmgr, err := connmgr.NewConnManager(0, *clients)
if err != nil {
This disables the resource manager and uses a semaphore to limit the number of new concurrent connection requests. You want to limit this since there's a pending queue of connections before they're accepted in go-libp2p and to avoid a dos vector we cap this queue. If you don't accept from the queue, then future connections will be dropped.
again, you probably don't want to use a null resource manager in production. And in a real network you aren't going to be getting a ton of concurrent requests at once (unless you're getting dos'd or you know what you're doing).
Thank you very much @Marco for this suggestion..
Having pursued it, I've discovered the following:
Replacing the ResourceManager by the null resource manager and a limiting semaphore as you suggest makes a huge difference.
In the libp2p triage discussion last night
Marten Seeman confirmed that there is a fair amount of crypto involved in the quic handshake. This is required to accept a connection. @Marco stated that the performance on a single machine (i.e. connecting to localhost) is much different to that across a real network. The latter would suffer from latencies etc. that would even out some of the peak loading experienced on a single host.
The team therefore propose to close this issue @SOVLOOKUP at present. They would be happy to reopen it should problems arise in production, and a more detailed use case be provided.
When the number of connected peers increases to 120+, node will refuse new connections with an error:
The error can be reproduced with:https://github.com/SOVLOOKUP/yunos/tree/libp2p
Version Information