libp2p / go-libp2p

libp2p implementation in Go
MIT License
6.08k stars 1.07k forks source link

Error `failed to negotiate security protocol` #2339

Closed iulianpascalau closed 1 year ago

iulianpascalau commented 1 year ago

Hello, migrating from

github.com/libp2p/go-libp2p v0.14.3
github.com/libp2p/go-libp2p-core v0.8.6

to

github.com/libp2p/go-libp2p v0.22.0

surfaced a potential bug that was found mostly by coincidence. First, we are instantiating the host in golang as follows:

address := fmt.Sprintf(args.ListenAddress+"%d", port)
opts := []libp2p.Option{
    libp2p.ListenAddrStrings(address),
    libp2p.Identity(p2pPrivateKey),
}

h, err := libp2p.New(opts...)
if err != nil {
..... 

where args.ListenAddress is a const set to "/ip4/0.0.0.0/tcp/"

The bug happens when we call the host's method Connect(ctx context.Context, pi peer.AddrInfo) error to try to manually connect to a peer. If the host managed to do the "NAT hole punching", the connection attempt succeeds, otherwise, it will fail with the following error:

failed to dial 16Uiu2HAm1RkrG96rvXj3AU6aFX4WdiAankVg8ttW8KpLuEwW1FUt:
  * [/ip4/[IP]/tcp/10000] failed to negotiate security protocol: read tcp4 192.168.30.176:10001->[IP]:10000: read: connection reset by peer

We've observed the coincidence between the peer-bounded interfaces and the appearance of the above-mentioned error. So, whenever we see bounded interfaces like:

/ip4/192.168.30.176/tcp/10001/p2p/16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY
/ip4/127.0.0.1/tcp/10001/p2p/16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY

the error appears, but when we see this:

/ip4/192.168.30.176/tcp/10001/p2p/16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY 
/ip4/127.0.0.1/tcp/10001/p2p/16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY      
/ip4/[public IP 1]/tcp/10001/p2p/16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY  
/ip4/[public IP 2]/tcp/10001/p2p/16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY 

the connection & peer discovery through kad-dht works.

Enabling the log subsystem on libp2p packages for the node that should have accepted the connection, we've observed the following log lines after the node initiating the connection bound on the public interfaces

2023-06-06T17:58:20.495Z    DEBUG   upgrader    upgrader/listener.go:133    listener <stream.Listener[TCP] /ip4/0.0.0.0/tcp/10000> accepted connection: <stream.Conn[TCP] /ip4/[IP]/tcp/10000 (16Uiu2HAm1RkrG96rvXj3AU6aFX4WdiAankVg8ttW8KpLuEwW1FUt) <-> /ip4/[public IP 1]/tcp/10001 (16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY)>
2023-06-06T17:58:20.495Z    DEBUG   swarm2  swarm/swarm.go:336  [16Uiu2HAm1RkrG96rvXj3AU6aFX4WdiAankVg8ttW8KpLuEwW1FUt] opening stream to peer [16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY]
2023-06-06T17:58:20.541Z    DEBUG   net/identify    identify/id.go:407  /ipfs/id/1.0.0 sent message to 16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY /ip4/[public IP 1]/tcp/10001
2023-06-06T17:58:20.542Z    DEBUG   net/identify    identify/id.go:439  /ipfs/id/1.0.0 received message from 16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY /ip4/[public IP 1]/tcp/10001
2023-06-06T17:58:20.542Z    DEBUG   net/identify    identify/id.go:635  16Uiu2HAm1RkrG96rvXj3AU6aFX4WdiAankVg8ttW8KpLuEwW1FUt received listen addrs for 16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY: [/ip4/192.168.30.176/tcp/10001]
2023-06-06T17:58:20.542Z    DEBUG   dht go-libp2p-kad-dht@v0.18.0/dht.go:646    peer found  {"peer": "16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY"}
2023-06-06T17:58:20.542Z    DEBUG   pubsub  go-libp2p-pubsub@v0.8.1/gossipsub.go:518    PEERUP: Add new peer 16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY using /meshsub/1.1.0
2023-06-06T17:58:20.588Z    DEBUG   dht go-libp2p-kad-dht@v0.18.0/dht.go:646    peer found  {"peer": "16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY"}
2023-06-06T17:58:20.588Z    DEBUG   dht go-libp2p-kad-dht@v0.18.0/dht_net.go:116    handling message    {"from": "16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY", "type": 4, "key": "ACUIAhIhAswnJcVeeBhauhvyBh1vuEUKGBnyf0YxSTcx4VTdVQ+3"}
2023-06-06T17:58:20.588Z    DEBUG   dht go-libp2p-kad-dht@v0.18.0/dht_net.go:133    handled message {"from": "16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY", "type": 4, "key": "ACUIAhIhAswnJcVeeBhauhvyBh1vuEUKGBnyf0YxSTcx4VTdVQ+3", "time": 0.000192637}
2023-06-06T17:58:20.588Z    DEBUG   dht go-libp2p-kad-dht@v0.18.0/dht_net.go:159    responded to message    {"from": "16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY", "type": 4, "key": "ACUIAhIhAswnJcVeeBhauhvyBh1vuEUKGBnyf0YxSTcx4VTdVQ+3", "time": 0.000223246}
2023-06-06T17:58:25.175Z    DEBUG   net/identify    identify/id.go:439  /ipfs/id/push/1.0.0 received message from 16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY /ip4/[public IP 1]/tcp/10001
2023-06-06T17:58:25.175Z    DEBUG   net/identify    identify/id.go:635  16Uiu2HAm1RkrG96rvXj3AU6aFX4WdiAankVg8ttW8KpLuEwW1FUt received listen addrs for 16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY: [/ip4/192.168.30.176/tcp/10001 /ip4/[public IP 1]/tcp/10001 /ip4/[IP1]/tcp/10001]
2023-06-06T17:58:27.004Z    DEBUG   dht go-libp2p-kad-dht@v0.18.0/dht.go:646    peer found  {"peer": "16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY"}
2023-06-06T17:58:27.004Z    DEBUG   dht go-libp2p-kad-dht@v0.18.0/dht_net.go:116    handling message    {"from": "16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY", "type": 4, "key": "EiAAAcv5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="}
2023-06-06T17:58:27.005Z    DEBUG   dht go-libp2p-kad-dht@v0.18.0/dht_net.go:133    handled message {"from": "16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY", "type": 4, "key": "EiAAAcv5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==", "time": 0.000260002}
2023-06-06T17:58:27.005Z    DEBUG   dht go-libp2p-kad-dht@v0.18.0/dht_net.go:159    responded to message    {"from": "16Uiu2HAm9AbmmP1RDCbACoBFNToFWho6u1Ef3tZ2DyiURUJg4hyY", "type": 4, "key": "EiAAAcv5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==", "time": 0.000291781}

Thanks and sorry for the long wall of text

Version Information
bufio
bytes
compress/flate
compress/gzip
compress/zlib
container/list
context
crypto
crypto/aes
crypto/cipher
crypto/des
crypto/dsa
crypto/ecdh
crypto/ecdsa
crypto/ed25519
crypto/elliptic
crypto/hmac
crypto/internal/alias
crypto/internal/bigmod
crypto/internal/boring
crypto/internal/boring/bbig
crypto/internal/boring/sig
crypto/internal/edwards25519
crypto/internal/edwards25519/field
crypto/internal/nistec
crypto/internal/nistec/fiat
crypto/internal/randutil
crypto/md5
crypto/rand
crypto/rc4
crypto/rsa
crypto/sha1
crypto/sha256
crypto/sha512
crypto/subtle
crypto/tls
crypto/x509
crypto/x509/pkix
database/sql
database/sql/driver
embed
encoding
encoding/asn1
encoding/base32
encoding/base64
encoding/binary
encoding/hex
encoding/json
encoding/pem
encoding/xml
errors
expvar
flag
fmt
github.com/benbjohnson/clock
github.com/beorn7/perks/quantile
github.com/cespare/xxhash/v2
github.com/cheekybits/genny/generic
github.com/containerd/cgroups
github.com/containerd/cgroups/stats/v1
github.com/coreos/go-systemd/v22/dbus
github.com/davecgh/go-spew/spew
github.com/davidlazar/go-crypto/salsa20
github.com/decred/dcrd/dcrec/secp256k1/v4
github.com/decred/dcrd/dcrec/secp256k1/v4/ecdsa
github.com/denisbrodbeck/machineid
github.com/docker/go-units
github.com/elastic/gosigar
github.com/flynn/noise
github.com/francoispqt/gojay
github.com/godbus/dbus/v5
github.com/gogo/protobuf/gogoproto
github.com/gogo/protobuf/io
github.com/gogo/protobuf/proto
github.com/gogo/protobuf/protoc-gen-gogo/descriptor
github.com/golang/protobuf/proto
github.com/golang/protobuf/ptypes/timestamp
github.com/google/gopacket/routing
github.com/google/uuid
github.com/gorilla/websocket
github.com/hashicorp/errwrap
github.com/hashicorp/go-multierror
github.com/hashicorp/golang-lru
github.com/hashicorp/golang-lru/simplelru
github.com/huin/goupnp
github.com/huin/goupnp/dcps/internetgateway1
github.com/huin/goupnp/dcps/internetgateway2
github.com/huin/goupnp/httpu
github.com/huin/goupnp/scpd
github.com/huin/goupnp/soap
github.com/huin/goupnp/ssdp
github.com/ipfs/go-cid
github.com/ipfs/go-datastore
github.com/ipfs/go-datastore/autobatch
github.com/ipfs/go-datastore/query
github.com/ipfs/go-datastore/sync
github.com/ipfs/go-ipfs-util
github.com/ipfs/go-ipns
github.com/ipfs/go-ipns/pb
github.com/ipfs/go-log
github.com/ipfs/go-log/tracer
github.com/ipfs/go-log/tracer/wire
github.com/ipfs/go-log/v2
github.com/ipfs/go-log/writer
github.com/ipld/go-ipld-prime
github.com/ipld/go-ipld-prime/codec/dagcbor
github.com/ipld/go-ipld-prime/linking/cid
github.com/ipld/go-ipld-prime/multicodec
github.com/ipld/go-ipld-prime/node/basic
github.com/ipld/go-ipld-prime/node/mixins
github.com/jackpal/go-nat-pmp
github.com/jbenet/go-temp-err-catcher
github.com/jbenet/goprocess
github.com/jbenet/goprocess/context
github.com/klauspost/compress
github.com/klauspost/compress/fse
github.com/klauspost/compress/huff0
github.com/klauspost/compress/internal/snapref
github.com/klauspost/compress/zstd
github.com/klauspost/compress/zstd/internal/xxhash
github.com/klauspost/cpuid/v2
github.com/koron/go-ssdp
github.com/libp2p/go-buffer-pool
github.com/libp2p/go-cidranger
github.com/libp2p/go-cidranger/net
github.com/libp2p/go-flow-metrics
github.com/libp2p/go-libp2p
github.com/libp2p/go-libp2p-asn-util
github.com/libp2p/go-libp2p-core/crypto
github.com/libp2p/go-libp2p-core/peer
github.com/libp2p/go-libp2p-core/peerstore
github.com/libp2p/go-libp2p-kad-dht
github.com/libp2p/go-libp2p-kad-dht/internal
github.com/libp2p/go-libp2p-kad-dht/internal/config
github.com/libp2p/go-libp2p-kad-dht/internal/net
github.com/libp2p/go-libp2p-kad-dht/metrics
github.com/libp2p/go-libp2p-kad-dht/pb
github.com/libp2p/go-libp2p-kad-dht/providers
github.com/libp2p/go-libp2p-kad-dht/qpeerset
github.com/libp2p/go-libp2p-kad-dht/rtrefresh
github.com/libp2p/go-libp2p-kbucket
github.com/libp2p/go-libp2p-kbucket/keyspace
github.com/libp2p/go-libp2p-kbucket/peerdiversity
github.com/libp2p/go-libp2p-pubsub
github.com/libp2p/go-libp2p-pubsub/pb
github.com/libp2p/go-libp2p-record
github.com/libp2p/go-libp2p-record/pb
github.com/libp2p/go-libp2p/config
github.com/libp2p/go-libp2p/core/canonicallog
github.com/libp2p/go-libp2p/core/connmgr
github.com/libp2p/go-libp2p/core/control
github.com/libp2p/go-libp2p/core/crypto
github.com/libp2p/go-libp2p/core/crypto/pb
github.com/libp2p/go-libp2p/core/discovery
github.com/libp2p/go-libp2p/core/event
github.com/libp2p/go-libp2p/core/host
github.com/libp2p/go-libp2p/core/internal/catch
github.com/libp2p/go-libp2p/core/introspection
github.com/libp2p/go-libp2p/core/introspection/pb
github.com/libp2p/go-libp2p/core/metrics
github.com/libp2p/go-libp2p/core/network
github.com/libp2p/go-libp2p/core/peer
github.com/libp2p/go-libp2p/core/peer/pb
github.com/libp2p/go-libp2p/core/peerstore
github.com/libp2p/go-libp2p/core/pnet
github.com/libp2p/go-libp2p/core/protocol
github.com/libp2p/go-libp2p/core/record
github.com/libp2p/go-libp2p/core/record/pb
github.com/libp2p/go-libp2p/core/routing
github.com/libp2p/go-libp2p/core/sec
github.com/libp2p/go-libp2p/core/sec/insecure
github.com/libp2p/go-libp2p/core/sec/insecure/pb
github.com/libp2p/go-libp2p/core/transport
github.com/libp2p/go-libp2p/p2p/discovery/backoff
github.com/libp2p/go-libp2p/p2p/host/autonat
github.com/libp2p/go-libp2p/p2p/host/autonat/pb
github.com/libp2p/go-libp2p/p2p/host/autorelay
github.com/libp2p/go-libp2p/p2p/host/basic
github.com/libp2p/go-libp2p/p2p/host/blank
github.com/libp2p/go-libp2p/p2p/host/eventbus
github.com/libp2p/go-libp2p/p2p/host/peerstore
github.com/libp2p/go-libp2p/p2p/host/peerstore/pstoremem
github.com/libp2p/go-libp2p/p2p/host/pstoremanager
github.com/libp2p/go-libp2p/p2p/host/relaysvc
github.com/libp2p/go-libp2p/p2p/host/resource-manager
github.com/libp2p/go-libp2p/p2p/host/routed
github.com/libp2p/go-libp2p/p2p/muxer/muxer-multistream
github.com/libp2p/go-libp2p/p2p/muxer/yamux
github.com/libp2p/go-libp2p/p2p/net/conn-security-multistream
github.com/libp2p/go-libp2p/p2p/net/connmgr
github.com/libp2p/go-libp2p/p2p/net/mock
github.com/libp2p/go-libp2p/p2p/net/nat
github.com/libp2p/go-libp2p/p2p/net/pnet
github.com/libp2p/go-libp2p/p2p/net/reuseport
github.com/libp2p/go-libp2p/p2p/net/swarm
github.com/libp2p/go-libp2p/p2p/net/upgrader
github.com/libp2p/go-libp2p/p2p/protocol/circuitv1/pb
github.com/libp2p/go-libp2p/p2p/protocol/circuitv1/relay
github.com/libp2p/go-libp2p/p2p/protocol/circuitv2/client
github.com/libp2p/go-libp2p/p2p/protocol/circuitv2/pb
github.com/libp2p/go-libp2p/p2p/protocol/circuitv2/proto
github.com/libp2p/go-libp2p/p2p/protocol/circuitv2/relay
github.com/libp2p/go-libp2p/p2p/protocol/circuitv2/util
github.com/libp2p/go-libp2p/p2p/protocol/holepunch
github.com/libp2p/go-libp2p/p2p/protocol/holepunch/pb
github.com/libp2p/go-libp2p/p2p/protocol/identify
github.com/libp2p/go-libp2p/p2p/protocol/identify/pb
github.com/libp2p/go-libp2p/p2p/protocol/ping
github.com/libp2p/go-libp2p/p2p/security/noise
github.com/libp2p/go-libp2p/p2p/security/noise/pb
github.com/libp2p/go-libp2p/p2p/security/tls
github.com/libp2p/go-libp2p/p2p/transport/quic
github.com/libp2p/go-libp2p/p2p/transport/tcp
github.com/libp2p/go-libp2p/p2p/transport/websocket
github.com/libp2p/go-msgio
github.com/libp2p/go-msgio/protoio
github.com/libp2p/go-nat
github.com/libp2p/go-netroute
github.com/libp2p/go-reuseport
github.com/libp2p/go-yamux/v3
github.com/lucas-clemente/quic-go
github.com/lucas-clemente/quic-go/internal/ackhandler
github.com/lucas-clemente/quic-go/internal/congestion
github.com/lucas-clemente/quic-go/internal/flowcontrol
github.com/lucas-clemente/quic-go/internal/handshake
github.com/lucas-clemente/quic-go/internal/logutils
github.com/lucas-clemente/quic-go/internal/protocol
github.com/lucas-clemente/quic-go/internal/qerr
github.com/lucas-clemente/quic-go/internal/qtls
github.com/lucas-clemente/quic-go/internal/utils
github.com/lucas-clemente/quic-go/internal/wire
github.com/lucas-clemente/quic-go/logging
github.com/lucas-clemente/quic-go/qlog
github.com/lucas-clemente/quic-go/quicvarint
github.com/marten-seemann/qtls-go1-19
github.com/marten-seemann/tcp
github.com/mattn/go-isatty
github.com/matttproud/golang_protobuf_extensions/pbutil
github.com/miekg/dns
github.com/mikioh/tcpinfo
github.com/mikioh/tcpopt
github.com/minio/sha256-simd
github.com/mr-tron/base58/base58
github.com/multiformats/go-base32
github.com/multiformats/go-base36
github.com/multiformats/go-multiaddr
github.com/multiformats/go-multiaddr-dns
github.com/multiformats/go-multiaddr-fmt
github.com/multiformats/go-multiaddr/net
github.com/multiformats/go-multibase
github.com/multiformats/go-multicodec
github.com/multiformats/go-multihash
github.com/multiformats/go-multihash/core
github.com/multiformats/go-multihash/register/all
github.com/multiformats/go-multihash/register/blake2
github.com/multiformats/go-multihash/register/blake3
github.com/multiformats/go-multihash/register/miniosha256
github.com/multiformats/go-multihash/register/murmur3
github.com/multiformats/go-multihash/register/sha3
github.com/multiformats/go-multistream
github.com/multiformats/go-varint
github.com/multiversx/mx-chain-core-go/core
github.com/multiversx/mx-chain-core-go/core/atomic
github.com/multiversx/mx-chain-core-go/core/check
github.com/multiversx/mx-chain-core-go/core/random
github.com/multiversx/mx-chain-core-go/core/throttler
github.com/multiversx/mx-chain-core-go/data
github.com/multiversx/mx-chain-core-go/data/batch
github.com/multiversx/mx-chain-core-go/data/headerVersionData
github.com/multiversx/mx-chain-core-go/display
github.com/multiversx/mx-chain-core-go/hashing
github.com/multiversx/mx-chain-core-go/marshal
github.com/multiversx/mx-chain-core-go/storage
github.com/multiversx/mx-chain-crypto-go
github.com/multiversx/mx-chain-crypto-go/signing
github.com/multiversx/mx-chain-crypto-go/signing/secp256k1
github.com/multiversx/mx-chain-crypto-go/signing/secp256k1/singlesig
github.com/multiversx/mx-chain-logger-go
github.com/multiversx/mx-chain-logger-go/proto
github.com/multiversx/mx-chain-p2p-go
github.com/multiversx/mx-chain-p2p-go/config
github.com/multiversx/mx-chain-p2p-go/data
github.com/multiversx/mx-chain-p2p-go/debug
github.com/multiversx/mx-chain-p2p-go/integrationTests
github.com/multiversx/mx-chain-p2p-go/integrationTests/peerDisconnecting
github.com/multiversx/mx-chain-p2p-go/integrationTests/peerDiscovery
github.com/multiversx/mx-chain-p2p-go/integrationTests/peerDiscovery/kadDht
github.com/multiversx/mx-chain-p2p-go/integrationTests/pubsub
github.com/multiversx/mx-chain-p2p-go/issues/pubsub_349
github.com/multiversx/mx-chain-p2p-go/libp2p
github.com/multiversx/mx-chain-p2p-go/libp2p/connectionMonitor
github.com/multiversx/mx-chain-p2p-go/libp2p/crypto
github.com/multiversx/mx-chain-p2p-go/libp2p/disabled
github.com/multiversx/mx-chain-p2p-go/libp2p/discovery
github.com/multiversx/mx-chain-p2p-go/libp2p/discovery/factory
github.com/multiversx/mx-chain-p2p-go/libp2p/metrics
github.com/multiversx/mx-chain-p2p-go/libp2p/metrics/factory
github.com/multiversx/mx-chain-p2p-go/libp2p/networksharding
github.com/multiversx/mx-chain-p2p-go/libp2p/networksharding/factory
github.com/multiversx/mx-chain-p2p-go/libp2p/networksharding/sorting
github.com/multiversx/mx-chain-p2p-go/message
github.com/multiversx/mx-chain-p2p-go/messageCheck
github.com/multiversx/mx-chain-p2p-go/mock
github.com/multiversx/mx-chain-p2p-go/peersHolder
github.com/multiversx/mx-chain-p2p-go/peersHolder/connectionStringValidator
github.com/multiversx/mx-chain-p2p-go/rating
github.com/multiversx/mx-chain-storage-go/common
github.com/multiversx/mx-chain-storage-go/lrucache
github.com/multiversx/mx-chain-storage-go/lrucache/capacity
github.com/multiversx/mx-chain-storage-go/timecache
github.com/multiversx/mx-chain-storage-go/types
github.com/opencontainers/runtime-spec/specs-go
github.com/opentracing/opentracing-go
github.com/opentracing/opentracing-go/ext
github.com/opentracing/opentracing-go/log
github.com/pbnjay/memory
github.com/pelletier/go-toml
github.com/pkg/errors
github.com/pmezard/go-difflib/difflib
github.com/polydawn/refmt/cbor
github.com/polydawn/refmt/obj
github.com/polydawn/refmt/obj/atlas
github.com/polydawn/refmt/shared
github.com/polydawn/refmt/tok
github.com/prometheus/client_golang/prometheus
github.com/prometheus/client_golang/prometheus/internal
github.com/prometheus/client_model/go
github.com/prometheus/common/expfmt
github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg
github.com/prometheus/common/model
github.com/prometheus/procfs
github.com/prometheus/procfs/internal/fs
github.com/prometheus/procfs/internal/util
github.com/raulk/go-watchdog
github.com/spaolacci/murmur3
github.com/stretchr/testify/assert
github.com/stretchr/testify/require
github.com/whyrusleeping/go-keyspace
github.com/whyrusleeping/timecache
go.opencensus.io/internal/tagencoding
go.opencensus.io/metric/metricdata
go.opencensus.io/metric/metricproducer
go.opencensus.io/resource
go.opencensus.io/stats
go.opencensus.io/stats/internal
go.opencensus.io/stats/view
go.opencensus.io/tag
go.uber.org/atomic
go.uber.org/multierr
go.uber.org/zap
go.uber.org/zap/buffer
go.uber.org/zap/internal/bufferpool
go.uber.org/zap/internal/color
go.uber.org/zap/internal/exit
go.uber.org/zap/zapcore
go/token
golang.org/x/crypto/blake2b
golang.org/x/crypto/blake2s
golang.org/x/crypto/chacha20
golang.org/x/crypto/chacha20poly1305
golang.org/x/crypto/cryptobyte
golang.org/x/crypto/cryptobyte/asn1
golang.org/x/crypto/curve25519
golang.org/x/crypto/curve25519/internal/field
golang.org/x/crypto/hkdf
golang.org/x/crypto/internal/alias
golang.org/x/crypto/internal/poly1305
golang.org/x/crypto/salsa20/salsa
golang.org/x/crypto/sha3
golang.org/x/net/bpf
golang.org/x/net/internal/iana
golang.org/x/net/internal/socket
golang.org/x/net/ipv4
golang.org/x/net/ipv6
golang.org/x/sync/errgroup
golang.org/x/sys/cpu
golang.org/x/sys/unix
google.golang.org/protobuf/encoding/prototext
google.golang.org/protobuf/encoding/protowire
google.golang.org/protobuf/internal/descfmt
google.golang.org/protobuf/internal/descopts
google.golang.org/protobuf/internal/detrand
google.golang.org/protobuf/internal/encoding/defval
google.golang.org/protobuf/internal/encoding/messageset
google.golang.org/protobuf/internal/encoding/tag
google.golang.org/protobuf/internal/encoding/text
google.golang.org/protobuf/internal/errors
google.golang.org/protobuf/internal/filedesc
google.golang.org/protobuf/internal/filetype
google.golang.org/protobuf/internal/flags
google.golang.org/protobuf/internal/genid
google.golang.org/protobuf/internal/impl
google.golang.org/protobuf/internal/order
google.golang.org/protobuf/internal/pragma
google.golang.org/protobuf/internal/set
google.golang.org/protobuf/internal/strs
google.golang.org/protobuf/internal/version
google.golang.org/protobuf/proto
google.golang.org/protobuf/reflect/protodesc
google.golang.org/protobuf/reflect/protoreflect
google.golang.org/protobuf/reflect/protoregistry
google.golang.org/protobuf/runtime/protoiface
google.golang.org/protobuf/runtime/protoimpl
google.golang.org/protobuf/types/descriptorpb
google.golang.org/protobuf/types/known/timestamppb
gopkg.in/yaml.v3
hash
hash/adler32
hash/crc32
hash/fnv
internal/abi
internal/bytealg
internal/coverage/rtcov
internal/cpu
internal/fmtsort
internal/goarch
internal/godebug
internal/goexperiment
internal/goos
internal/intern
internal/itoa
internal/nettrace
internal/oserror
internal/poll
internal/race
internal/reflectlite
internal/safefilepath
internal/singleflight
internal/syscall/execenv
internal/syscall/unix
internal/sysinfo
internal/testlog
internal/unsafeheader
io
io/fs
io/ioutil
log
lukechampine.com/blake3
math
math/big
math/bits
math/rand
mime
mime/multipart
mime/quotedprintable
net
net/http
net/http/httptest
net/http/httptrace
net/http/internal
net/http/internal/ascii
net/http/internal/testcert
net/netip
net/textproto
net/url
os
os/exec
os/signal
os/user
path
path/filepath
reflect
regexp
regexp/syntax
runtime
runtime/debug
runtime/internal/atomic
runtime/internal/math
runtime/internal/sys
runtime/internal/syscall
runtime/metrics
runtime/pprof
runtime/trace
sort
strconv
strings
sync
sync/atomic
syscall
testing
text/tabwriter
time
unicode
unicode/utf16
unicode/utf8
unsafe
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/crypto/chacha20poly1305
vendor/golang.org/x/crypto/cryptobyte
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/hkdf
vendor/golang.org/x/crypto/internal/alias
vendor/golang.org/x/crypto/internal/poly1305
vendor/golang.org/x/net/dns/dnsmessage
vendor/golang.org/x/net/http/httpguts
vendor/golang.org/x/net/http/httpproxy
vendor/golang.org/x/net/http2/hpack
vendor/golang.org/x/net/idna
vendor/golang.org/x/sys/cpu
vendor/golang.org/x/text/secure/bidirule
vendor/golang.org/x/text/transform
vendor/golang.org/x/text/unicode/bidi
vendor/golang.org/x/text/unicode/norm

marten-seemann commented 1 year ago

Does this also happen with the current version? v0.22 is pretty old.

iulianpascalau commented 1 year ago

Will try. Thanks for the suggestion.

iulianpascalau commented 1 year ago

Tested with

go-libp2p v0.27.5
go-libp2p-kad-dht v0.23.0
go-libp2p-kbucket v0.5.0
go-libp2p-pubsub v0.9.3
go-multiaddr v0.9.0

and the go compiler v1.20.3 and we've got the same results: while the host is not bound to the public interface, is not able to manually connect to another host.

marten-seemann commented 1 year ago

Your listen address is wrong, the port number is missing.

Other than that, the listening and the dialing code path don’t have a lot of overlap (other than reuseport, might be worth trying if disabling it helps). You could also inspect what’s going on on the wire using Wireshark.

I don’t really understand your point about holepunching. Hole punching requires you to have a connection to a relay server, at the very least. So if hole punching works, you were able to establish at least that one connection.

iulianpascalau commented 1 year ago

The code above might be a little confusing as the port number is always provided to the new host instance:

address := fmt.Sprintf(args.ListenAddress+"%d", port)
opts := []libp2p.Option{
    libp2p.ListenAddrStrings(address),

if args.ListenAddress is set to /ip4/0.0.0.0/tcp/ then the whole address var will hold something like /ip4/0.0.0.0/tcp/10001 if the provided port was 10001

The holepunching I was referring to was related to the NAT', UPnP feature. Since the host I'm trying to work on is behind a NAT, it will use the UPnP to negotiate a way to bind on the public interface(s) on a port given by the NAT. What strikes me is that using very old libp2p libs, this connection always works despite the fact that the node triggering the connection managed to bind on the public interface or not. Will try to narrow down this issue (maybe adding more prints or something).

Thanks once again for your time

iulianpascalau commented 1 year ago

yup, confirmed that tcp.DisableReuseport() fixed the issue. Since this setting will need to be set only on our seednodes, I guess I can make it configurable.

Thanks for the help, this issue can be closed.

marten-seemann commented 1 year ago

Great, thanks for confirming. Do you have any theory why reuseport is causing problems here?

iulianpascalau commented 1 year ago

No, not a clue why disabling the reuse port fixes the issue.

MarcoPolo commented 1 year ago

I wonder if this is router specific. Does reuseport confuse the router? Does it prevent upnp? Something else?