Giulio2002
commented
1 year ago Tried to downgrade to 0.25 as well but same issue there too
Closed Giulio2002 closed 1 year ago
Giulio2002
commented
1 year ago Tried to downgrade to 0.25 as well but same issue there too
MarcoPolo
commented
1 year ago We removed identify delta in v0.25:
I found a work around that seems to somehow fix this as long as I do not do any request which is calling host.RemoveStreamHandler("/p2p/id/delta/1.0.0")
I don't understand. A request is asking you to remove that stream handler? Or do you mean there are requests on the network for /p2p/id/delta/1.0.0 and as long as you handle them you don't get kicked out?
What if you had a no-op implementation for /p2p/id/delta/1.0.0, would that workaround the problem? This isn't a fix, just a way to debug the issue.
e.g.:
host.SetStreamHandler("/p2p/id/delta/1.0.0", func(s network.Stream) {
s.Close()
})I mean that removing that handler somehow makes it work again but only for gossip, if I do requests i start getting disconnected. NOTE: I am not sure if it is me banning somehow or them banning me, what I know is that after 8-9 hours of sending requests I lose all my peers and cannot reconnect again
Giulio2002
commented
1 year ago Could it be that the other clients of the network uses a lower libp2p version and that is causing issues, perhaps? because none of them are running the latests in any release. If you are wondering how I came out with the identify.IdDelta fixup was that I compared it to prysm (another ETH 2.0 client) and it seems that they did remove IdDelta. So i just hardcoded the value and it just worked for as long as I do not make any requests, before it just randomly used to disconnect. Also I am now testing your snippet.
Giulio2002
commented
1 year ago still dies with handler
MarcoPolo
commented
1 year ago I mean that removing that handler somehow makes it work again but only for gossip
This is strange. We don't have that handler registered so removing it should be a no-op?
MarcoPolo
commented
1 year ago Prysm is on go-libp2p v0.26.2 and we haven't gotten an issue from them yet. This might be an issue with your specific setup. Maybe some action is negatively affecting your gossipsub peer score?
Giulio2002
commented
1 year ago mmmh, i see... I will see what I can do, If I find something out I will write it here, you can close this issue if you wish btw :)
Giulio2002
commented
1 year ago Ok but hold up a second, this commit https://github.com/ledgerwatch/erigon/commit/d5bb42453acf0d5a1a1bba24b238310d86d59030 (do not worry this is just the dep upgrade) broke it for me with no change to the configuration at all and to my setup. Even if prysm has not found an issue yet, I do not see how this is expected behaviour for an upgrade to be breaking on any config. this commit is broken the one before is fine. i am very confused by this.
MarcoPolo
commented
1 year ago Can you run with GOLOG_LOG_LEVEL=debug on the working and non-working version and share the logs? You can email the logs privately to me if you like at the email on my GitHub profile.
MarcoPolo
commented
1 year ago I wonder what the difference is in those logs.
Giulio2002
commented
1 year ago Update: it seems that disbabling pubsub makes the request not kick me again, i think it is a libp2p-go pubsub issue. should I move it to their repo?
MarcoPolo
commented
1 year ago Yes please. Tag me there as well and reference this issue.
Giulio2002
commented
1 year ago btw i have the feelings that these logs are going to be long, should I cut them? It will take a few days as my application needs to run for 12 hours before dying
Giulio2002
commented
1 year ago here is a taste though of the broken version logs:
2023-03-27T21:36:00.376Z DEBUG swarm2 swarm/limiter.go:167 [limiter] executing dial; peer: 16Uiu2HAmUfAvMBgqb9JhzXKj2SbXJhiHiYzjY3LC2KbUVJ7w2w2Z; addr: /ip4/17>
2023-03-27T21:36:00.376Z DEBUG swarm2 swarm/swarm_dial.go:485 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs swarm dialing 16Uiu2HAmUfAvMBgqb9JhzXKj2SbXJhi>
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/limiter.go:73 [limiter] freeing FD token; waiting: 0; consuming: 34
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/limiter.go:100 [limiter] freeing peer token; peer 16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6hVHA9YgqDxHagH; addr: /ip4>
2023-03-27T21:36:00.377Z DEBUG net/identify identify/id.go:388 error opening identify stream {"peer": "16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6hVHA9YgqDxH>
2023-03-27T21:36:00.377Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6hVHA9YgqDxHagH: session shut down
2023-03-27T21:36:00.377Z DEBUG basichost basic/basic_host.go:735 host 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs finished dialing 16Uiu2HAkutpUFRH>
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/swarm.go:362 [16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs] opening stream to peer [16Uiu2HAkutpUFRHunr8>
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/limiter.go:201 [limiter] clearing all peer dials: 16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6hVHA9YgqDxHagH
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/swarm_dial.go:243 dialing peer {"from": "16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs", "to": "16Uiu2HAkut>
2023-03-27T21:36:00.377Z DEBUG basichost basic/basic_host.go:718 host 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs dialing 16Uiu2HAkutpUFRHunr8E1sE1>
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/swarm_dial.go:243 dialing peer {"from": "16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs", "to": "16Uiu2HAkut>
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/limiter.go:193 [limiter] adding a dial job through limiter: /ip4/162.55.235.29/tcp/9000
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/limiter.go:161 [limiter] taking FD token: peer: 16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6hVHA9YgqDxHagH; addr: /ip4/1>
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/limiter.go:167 [limiter] executing dial; peer: 16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6hVHA9YgqDxHagH; addr: /ip4/16>
2023-03-27T21:36:00.377Z DEBUG swarm2 swarm/swarm_dial.go:485 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs swarm dialing 16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE>
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/limiter.go:73 [limiter] freeing FD token; waiting: 0; consuming: 34
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/limiter.go:100 [limiter] freeing peer token; peer 16Uiu2HAmUfAvMBgqb9JhzXKj2SbXJhiHiYzjY3LC2KbUVJ7w2w2Z; addr: /ip4>
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/limiter.go:201 [limiter] clearing all peer dials: 16Uiu2HAmUfAvMBgqb9JhzXKj2SbXJhiHiYzjY3LC2KbUVJ7w2w2Z
2023-03-27T21:36:00.379Z DEBUG net/identify identify/id.go:388 error opening identify stream {"peer": "16Uiu2HAmUfAvMBgqb9JhzXKj2SbXJhiHiYzjY3LC2KbUVJ7w2>
2023-03-27T21:36:00.379Z DEBUG basichost basic/basic_host.go:718 host 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs dialing 16Uiu2HAmUfAvMBgqb9JhzXKj>
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/swarm_dial.go:243 dialing peer {"from": "16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs", "to": "16Uiu2HAmUf>
2023-03-27T21:36:00.379Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmUfAvMBgqb9JhzXKj2SbXJhiHiYzjY3LC2KbUVJ7w2w2Z: session shut down
2023-03-27T21:36:00.379Z DEBUG basichost basic/basic_host.go:735 host 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs finished dialing 16Uiu2HAmUfAvMBg>
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/limiter.go:193 [limiter] adding a dial job through limiter: /ip4/176.9.22.143/tcp/9000
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/swarm.go:362 [16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs] opening stream to peer [16Uiu2HAmUfAvMBgqb9J>
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/limiter.go:161 [limiter] taking FD token: peer: 16Uiu2HAmUfAvMBgqb9JhzXKj2SbXJhiHiYzjY3LC2KbUVJ7w2w2Z; addr: /ip4/1>
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/limiter.go:167 [limiter] executing dial; peer: 16Uiu2HAmUfAvMBgqb9JhzXKj2SbXJhiHiYzjY3LC2KbUVJ7w2w2Z; addr: /ip4/17>
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/swarm_dial.go:243 dialing peer {"from": "16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs", "to": "16Uiu2HAmUf>
2023-03-27T21:36:00.379Z DEBUG swarm2 swarm/swarm_dial.go:485 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs swarm dialing 16Uiu2HAmUfAvMBgqb9JhzXKj2SbXJhi>
2023-03-27T21:36:00.380Z DEBUG swarm2 swarm/limiter.go:73 [limiter] freeing FD token; waiting: 0; consuming: 34
2023-03-27T21:36:00.380Z DEBUG swarm2 swarm/limiter.go:100 [limiter] freeing peer token; peer 16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6hVHA9YgqDxHagH; addr: /ip4>
2023-03-27T21:36:00.380Z DEBUG swarm2 swarm/swarm.go:362 [16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs] opening stream to peer [16Uiu2HAkutpUFRHunr8>
2023-03-27T21:36:00.380Z DEBUG pubsub go-libp2p-pubsub@v0.9.0/comm.go:117 opening new stream to peer: session shut down16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6>
2023-03-27T21:36:00.380Z DEBUG swarm2 swarm/swarm.go:362 [16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs] opening stream to peer [16Uiu2HAkutpUFRHunr8>
2023-03-27T21:36:00.380Z INFO net/identify identify/id.go:399 failed negotiate identify protocol with peer {"peer": "16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5g>
2023-03-27T21:36:00.380Z DEBUG swarm2 swarm/swarm_dial.go:243 dialing peer {"from": "16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs", "to": "16Uiu2HAkut>
2023-03-27T21:36:00.380Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6hVHA9YgqDxHagH: session shut down
UN92XjaJntZkQE8q2023-03-27T21:36:00.380Z DEBUG swarm2 swarm/limiter.go:201 [limiter] clearing all peer dials: 16Uiu2HAkutpUFRHunr8E1sE1TjT9MFE5gFe6T6hVHA9YgqDx4Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb
2023-03-28T09:21:32.093Z DEBUG basichost basic/basic_host.go:718 host 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs dialing 16Uiu2HAkwchwTCd44Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb
2023-03-28T09:21:32.093Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAkwchwTCd44Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb: session shut down
2023-03-28T09:21:32.093Z DEBUG swarm2 swarm/swarm.go:362 [16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs] opening stream to peer [16Uiu2HAkwchwTCd44Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb]
2023-03-28T09:21:32.093Z DEBUG swarm2 swarm/swarm_dial.go:243 dialing peer {"from": "16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs", "to": "16Uiu2HAkwchwTCd44Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb"}
2023-03-28T09:21:32.093Z DEBUG swarm2 swarm/swarm_dial.go:243 dialing peer {"from": "16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs", "to": "16Uiu2HAkwchwTCd44Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb"}
2023-03-28T09:21:32.093Z DEBUG swarm2 swarm/limiter.go:201 [limiter] clearing all peer dials: 16Uiu2HAkwchwTCd44Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb
2023-03-28T09:21:32.093Z DEBUG swarm2 swarm/limiter.go:193 [limiter] adding a dial job through limiter: /ip4/65.109.144.95/tcp/9000
2023-03-28T09:21:32.093Z DEBUG swarm2 swarm/limiter.go:161 [limiter] taking FD token: peer: 16Uiu2HAkwchwTCd44Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb; addr: /ip4/65.109.144.95/tcp/9000; prev consuming: 65
2023-03-28T09:21:32.093Z DEBUG swarm2 swarm/limiter.go:167 [limiter] executing dial; peer: 16Uiu2HAkwchwTCd44Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb; addr: /ip4/65.109.144.95/tcp/9000; FD consuming: 66; waiting: 0
2023-03-28T09:21:32.093Z DEBUG swarm2 swarm/swarm_dial.go:485 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs swarm dialing 16Uiu2HAkwchwTCd44Dnwuy1otS1tQn4AdRpuzGDhpRht1ZQkqodb /ip4/65.109.144.95/tcp/9000
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/limiter.go:73 [limiter] freeing FD token; waiting: 0; consuming: 66
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/limiter.go:100 [limiter] freeing peer token; peer 16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7; addr: /ip4/98.39.65.125/tcp/9504; active for peer: 1; waiting on peer limit: 0
2023-03-28T09:21:32.095Z DEBUG basichost basic/basic_host.go:735 host 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs finished dialing 16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7
2023-03-28T09:21:32.095Z DEBUG basichost basic/basic_host.go:718 host 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs dialing 16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7
2023-03-28T09:21:32.095Z DEBUG net/identify identify/id.go:388 error opening identify stream {"peer": "16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7", "error": "session shut down"}
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/swarm_dial.go:243 dialing peer {"from": "16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs", "to": "16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7"}
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/limiter.go:201 [limiter] clearing all peer dials: 16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7
2023-03-28T09:21:32.095Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7: session shut down
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/swarm.go:362 [16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs] opening stream to peer [16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7]
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/swarm_dial.go:243 dialing peer {"from": "16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs", "to": "16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7"}
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/limiter.go:193 [limiter] adding a dial job through limiter: /ip4/98.39.65.125/tcp/9504
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/limiter.go:161 [limiter] taking FD token: peer: 16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7; addr: /ip4/98.39.65.125/tcp/9504; prev consuming: 65
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/limiter.go:167 [limiter] executing dial; peer: 16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7; addr: /ip4/98.39.65.125/tcp/9504; FD consuming: 66; waiting: 0
2023-03-28T09:21:32.095Z DEBUG swarm2 swarm/swarm_dial.go:485 16Uiu2HAmS1pR12jQg4LygeZFK148mTy25hKCUYcoiYaX8Yf3ZDZs swarm dialing 16Uiu2HAmGiyWEJj8L2kBND1DkZBoq1z28kKNLUPEynGU11koYvb7 /ip4/98.39.65.125/tcp/9504
post death. full log file is 29 GB so probably it is better cut.
MarcoPolo
commented
1 year ago Anything stand out to you? Maybe log at a warn level instead?
Giulio2002
commented
1 year ago warn:
2023-03-28T20:13:42.441Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAkvYZx4VmB2TDYZnWLVdtLccBgsYwras2E9wq68YXBNf6U: stream reset
2023-03-28T20:13:42.509Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:43722->135.181.129.149:9000: use of closed network connection
2023-03-28T20:13:42.509Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: session shut down
2023-03-28T20:13:42.562Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:53456->149.28.16.106:9002: use of closed network connection
2023-03-28T20:13:42.562Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmMuE2Utb9M24vNL4fgUuwu83xC4BnNASaoov6Mp1MuRqJ: stream reset
2023-03-28T20:13:42.611Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: session shut down
2023-03-28T20:13:42.611Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: session shut down
2023-03-28T20:13:42.713Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: session shut down
2023-03-28T20:13:42.815Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: session shut down
2023-03-28T20:13:42.815Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: stream reset
2023-03-28T20:13:42.820Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmQ9K7QATL5iUXKrSQQXcrVo78Dg2kkMnEMkh9GTZKHkCJ: session shut down
2023-03-28T20:13:42.893Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: session shut down
2023-03-28T20:13:42.893Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmMcFb2knmUUxSPRcWX96Qdu6xbU8FWnRxHfTVLxGU3JpK: session shut down
2023-03-28T20:13:42.916Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: session shut down
2023-03-28T20:13:42.916Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: stream reset
2023-03-28T20:13:43.018Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:43784->135.181.129.149:9000: use of closed network connection
2023-03-28T20:13:43.018Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: stream reset
2023-03-28T20:13:43.120Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: session shut down
2023-03-28T20:13:43.192Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmQ9K7QATL5iUXKrSQQXcrVo78Dg2kkMnEMkh9GTZKHkCJ: stream reset
2023-03-28T20:13:43.230Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: session shut down
2023-03-28T20:13:43.332Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:43820->135.181.129.149:9000: use of closed network connection
2023-03-28T20:13:43.332Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: stream reset
2023-03-28T20:13:43.349Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: session shut down
2023-03-28T20:13:43.349Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmMcFb2knmUUxSPRcWX96Qdu6xbU8FWnRxHfTVLxGU3JpK: session shut down
2023-03-28T20:13:43.434Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: session shut down
2023-03-28T20:13:43.536Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: stream reset
2023-03-28T20:13:43.638Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:43854->135.181.129.149:9000: use of closed network connection
2023-03-28T20:13:43.638Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: stream reset
2023-03-28T20:13:43.646Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:53846->54.172.175.182:9000: use of closed network connection
2023-03-28T20:13:43.646Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmQ9K7QATL5iUXKrSQQXcrVo78Dg2kkMnEMkh9GTZKHkCJ: stream reset
2023-03-28T20:13:43.740Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: session shut down
2023-03-28T20:13:43.793Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:55748->108.39.88.12:9001: use of closed network connection
2023-03-28T20:13:43.793Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmMcFb2knmUUxSPRcWX96Qdu6xbU8FWnRxHfTVLxGU3JpK: stream reset
2023-03-28T20:13:43.842Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAm8ZBgh8HWRtxiSghUjmjEthR7E46uijcHMg5Uzf8RkdYx: session shut down
2023-03-28T20:13:44.012Z WARN net/identify identify/id.go:374 failed to identify 16Uiu2HAmQ9K7QATL5iUXKrSQQXcrVo78Dg2kkMnEMkh9GTZKHkCJ: stream resetfor broken
2023-03-28T20:17:15.443Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.446Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.446Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.449Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.449Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.449Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.453Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.453Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.453Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.457Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:37346->148.251.133.208:9000: use of closed network connection
2023-03-28T20:17:15.457Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.457Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.457Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.460Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.463Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.466Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.466Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.469Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:37384->148.251.133.208:9000: use of closed network connection
2023-03-28T20:17:15.469Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.469Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.469Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmBq9YbXpha8RqUpM55dQ8B7jJa8RhaqYb8jZERauWnci1: session shut down
2023-03-28T20:17:15.501Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:36574->34.91.62.48:13000: use of closed network connection
2023-03-28T20:17:15.777Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: session shut down
2023-03-28T20:17:15.777Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCtWopxoZgqykenbcESTMRtspSGWgJxefvP9MvfM8UFSU: session shut down
2023-03-28T20:17:15.781Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCir1J5dRM3DxfTKFND8indwtZgEraqKhXvhUJUTyksdc: session shut down
2023-03-28T20:17:16.133Z WARN net/identify identify/id.go:437 error reading identify message: stream reset
2023-03-28T20:17:16.133Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAm64nAebwsCtqiPt4ySuTB2yR2McMineQy4RcSoAumgH2N: stream reset
2023-03-28T20:17:16.144Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCir1J5dRM3DxfTKFND8indwtZgEraqKhXvhUJUTyksdc: session shut down
2023-03-28T20:17:16.144Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCir1J5dRM3DxfTKFND8indwtZgEraqKhXvhUJUTyksdc: session shut down
2023-03-28T20:17:16.503Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCir1J5dRM3DxfTKFND8indwtZgEraqKhXvhUJUTyksdc: session shut down
2023-03-28T20:17:16.625Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: session shut down
2023-03-28T20:17:16.625Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCtWopxoZgqykenbcESTMRtspSGWgJxefvP9MvfM8UFSU: session shut down
2023-03-28T20:17:16.746Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmLuzqEx653drpfYVhpwvQ4Yr1MAX5GLRg9Hm2oM5w3SZp: session shut down
2023-03-28T20:17:16.862Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCir1J5dRM3DxfTKFND8indwtZgEraqKhXvhUJUTyksdc: session shut down
2023-03-28T20:17:16.862Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCir1J5dRM3DxfTKFND8indwtZgEraqKhXvhUJUTyksdc: session shut down
2023-03-28T20:17:16.985Z WARN mplex go-mplex@v0.7.0/multiplex.go:271 error writing data: write tcp4 142.132.137.22:49538->198.44.136.104:56964: use of closed network connection
2023-03-28T20:17:16.985Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCtWopxoZgqykenbcESTMRtspSGWgJxefvP9MvfM8UFSU: session shut down
2023-03-28T20:17:16.985Z WARN net/identify identify/id.go:334 failed to identify 16Uiu2HAmCtWopxoZgqykenbcESTMRtspSGWgJxefvP9MvfM8UFSU: session shut downfor not broken
Giulio2002
commented
1 year ago same thing pretty much. i think I will try to test all pubsub version until implosion in beetwen
vyzo
commented
1 year ago look, this is not a pubsub issue, it is a libp2p regression that breaks pubsub.
Giulio2002
commented
1 year ago Anyway at this point I will just use Prysm config where possible even though It is quite messy. will leave open for closure in case.
MarcoPolo
commented
1 year ago look, this is not a pubsub issue, it is a libp2p regression that breaks pubsub.
What regression?
vyzo
commented
1 year ago the only thing that has changed is libp2p, which keeps changing too fast for not particularly good reasons and without proper regression testing... and now that network is broken.
MarcoPolo
commented
1 year ago @Giulio2002 if you want to hop on a call to pair debug this I’m happy to do so. We can coordinate on filecoin slack: https://filecoin.io/slack
Giulio2002
commented
1 year ago DMed you, text me whenever... i think we are on different timezones though
MarcoPolo
commented
1 year ago Per our out of band discussion, this isn't losing peers. Rather the "subscription to gossip seems not to count new peers after a while". I think we can close this issue, but can maybe reopen another issue when we root cause the issue.
Description
Hello, I am from the Erigon team, one of the Ethereum clients, I am trying to build a consensus client and I am having issues after upgrading libp2p to 26.2. What seems to happen is that after the upgrade I lose all the peers I have and get kicked out of the network eventually. I found a work around that seems to somehow fix this as long as I do not do any request which is calling
host.RemoveStreamHandler("/p2p/id/delta/1.0.0"). If I do that when I only subscribe and publish to gossip it seems to not give any issue, however if I repeatedly make requests to the network, I will get kicked out from the network (or at least lose all peers and be unable to reconnect).I get kicked out after a while from the ETH 2.0 network after upgrading libp2p version.