libp2p / go-libp2p

libp2p implementation in Go
MIT License
6.13k stars 1.08k forks source link

Peerstore reports empty protocolID list #2643

Open juligasa opened 1 year ago

juligasa commented 1 year ago

In our network, we identify peers talking hypermedia protocol by checking if that protocol is present in the list of protocols when connecting to a peer. We get the list of protocols by doing <caller>.Peerstore().GetProtocols(<called_pid>) (we could be using FirstSupportedProtocol but for the sake of this issue, getting all supported protocols is best) after connecting with a peer on the network. The usual response is like: [/ipfs/lan/kad/1.0.0 /hypermedia/0.2.0 /ipfs/bitswap /ipfs/bitswap/1.0.0 /ipfs/bitswap/1.1.0 /ipfs/bitswap/1.2.0 /ipfs/id/push/1.0.0 /ipfs/id/1.0.0 /ipfs/ping/1.0.0 /libp2p/circuit/relay/0.2.0/stop] However, some peers report no protocols at all [] (as if we were issuing RemoveProtocols) and the only way to make them work properly again is either restarting the called node or restarting the caller node. Note that while they are in this protocolless state, any subsequent calls to NewStream() will fail (with the error failed to negotiate protocol: context deadline exceeded), which is the real blocker in our case. After restarting the caller, a new random set of peers will face the same issue. In unitary tests, this seems to work. Any idea on what's going on?

Version Information

mintter
cloud.google.com/go v0.105.0
cloud.google.com/go/accessapproval v1.5.0
cloud.google.com/go/accesscontextmanager v1.4.0
cloud.google.com/go/aiplatform v1.27.0
cloud.google.com/go/analytics v0.12.0
cloud.google.com/go/apigateway v1.4.0
cloud.google.com/go/apigeeconnect v1.4.0
cloud.google.com/go/appengine v1.5.0
cloud.google.com/go/area120 v0.6.0
cloud.google.com/go/artifactregistry v1.9.0
cloud.google.com/go/asset v1.10.0
cloud.google.com/go/assuredworkloads v1.9.0
cloud.google.com/go/automl v1.8.0
cloud.google.com/go/baremetalsolution v0.4.0
cloud.google.com/go/batch v0.4.0
cloud.google.com/go/beyondcorp v0.3.0
cloud.google.com/go/bigquery v1.44.0
cloud.google.com/go/billing v1.7.0
cloud.google.com/go/binaryauthorization v1.4.0
cloud.google.com/go/certificatemanager v1.4.0
cloud.google.com/go/channel v1.9.0
cloud.google.com/go/cloudbuild v1.4.0
cloud.google.com/go/clouddms v1.4.0
cloud.google.com/go/cloudtasks v1.8.0
cloud.google.com/go/compute v1.15.1
cloud.google.com/go/compute/metadata v0.2.3
cloud.google.com/go/contactcenterinsights v1.4.0
cloud.google.com/go/container v1.7.0
cloud.google.com/go/containeranalysis v0.6.0
cloud.google.com/go/datacatalog v1.8.0
cloud.google.com/go/dataflow v0.7.0
cloud.google.com/go/dataform v0.5.0
cloud.google.com/go/datafusion v1.5.0
cloud.google.com/go/datalabeling v0.6.0
cloud.google.com/go/dataplex v1.4.0
cloud.google.com/go/dataproc v1.8.0
cloud.google.com/go/dataqna v0.6.0
cloud.google.com/go/datastore v1.10.0
cloud.google.com/go/datastream v1.5.0
cloud.google.com/go/deploy v1.5.0
cloud.google.com/go/dialogflow v1.19.0
cloud.google.com/go/dlp v1.7.0
cloud.google.com/go/documentai v1.10.0
cloud.google.com/go/domains v0.7.0
cloud.google.com/go/edgecontainer v0.2.0
cloud.google.com/go/errorreporting v0.3.0
cloud.google.com/go/essentialcontacts v1.4.0
cloud.google.com/go/eventarc v1.8.0
cloud.google.com/go/filestore v1.4.0
cloud.google.com/go/firestore v1.9.0
cloud.google.com/go/functions v1.9.0
cloud.google.com/go/gaming v1.8.0
cloud.google.com/go/gkebackup v0.3.0
cloud.google.com/go/gkeconnect v0.6.0
cloud.google.com/go/gkehub v0.10.0
cloud.google.com/go/gkemulticloud v0.4.0
cloud.google.com/go/gsuiteaddons v1.4.0
cloud.google.com/go/iam v0.8.0
cloud.google.com/go/iap v1.5.0
cloud.google.com/go/ids v1.2.0
cloud.google.com/go/iot v1.4.0
cloud.google.com/go/kms v1.6.0
cloud.google.com/go/language v1.8.0
cloud.google.com/go/lifesciences v0.6.0
cloud.google.com/go/logging v1.6.1
cloud.google.com/go/longrunning v0.3.0
cloud.google.com/go/managedidentities v1.4.0
cloud.google.com/go/maps v0.1.0
cloud.google.com/go/mediatranslation v0.6.0
cloud.google.com/go/memcache v1.7.0
cloud.google.com/go/metastore v1.8.0
cloud.google.com/go/monitoring v1.8.0
cloud.google.com/go/networkconnectivity v1.7.0
cloud.google.com/go/networkmanagement v1.5.0
cloud.google.com/go/networksecurity v0.6.0
cloud.google.com/go/notebooks v1.5.0
cloud.google.com/go/optimization v1.2.0
cloud.google.com/go/orchestration v1.4.0
cloud.google.com/go/orgpolicy v1.5.0
cloud.google.com/go/osconfig v1.10.0
cloud.google.com/go/oslogin v1.7.0
cloud.google.com/go/phishingprotection v0.6.0
cloud.google.com/go/policytroubleshooter v1.4.0
cloud.google.com/go/privatecatalog v0.6.0
cloud.google.com/go/pubsub v1.27.1
cloud.google.com/go/pubsublite v1.5.0
cloud.google.com/go/recaptchaenterprise/v2 v2.5.0
cloud.google.com/go/recommendationengine v0.6.0
cloud.google.com/go/recommender v1.8.0
cloud.google.com/go/redis v1.10.0
cloud.google.com/go/resourcemanager v1.4.0
cloud.google.com/go/resourcesettings v1.4.0
cloud.google.com/go/retail v1.11.0
cloud.google.com/go/run v0.3.0
cloud.google.com/go/scheduler v1.7.0
cloud.google.com/go/secretmanager v1.9.0
cloud.google.com/go/security v1.10.0
cloud.google.com/go/securitycenter v1.16.0
cloud.google.com/go/servicecontrol v1.5.0
cloud.google.com/go/servicedirectory v1.7.0
cloud.google.com/go/servicemanagement v1.5.0
cloud.google.com/go/serviceusage v1.4.0
cloud.google.com/go/shell v1.4.0
cloud.google.com/go/spanner v1.41.0
cloud.google.com/go/speech v1.9.0
cloud.google.com/go/storage v1.0.0
cloud.google.com/go/storagetransfer v1.6.0
cloud.google.com/go/talent v1.4.0
cloud.google.com/go/texttospeech v1.5.0
cloud.google.com/go/tpu v1.4.0
cloud.google.com/go/trace v1.4.0
cloud.google.com/go/translate v1.4.0
cloud.google.com/go/video v1.9.0
cloud.google.com/go/videointelligence v1.9.0
cloud.google.com/go/vision/v2 v2.5.0
cloud.google.com/go/vmmigration v1.3.0
cloud.google.com/go/vmwareengine v0.1.0
cloud.google.com/go/vpcaccess v1.5.0
cloud.google.com/go/webrisk v1.7.0
cloud.google.com/go/websecurityscanner v1.4.0
cloud.google.com/go/workflows v1.9.0
crawshaw.io/iox v0.0.0-20181124134642-c51c3df30797
crawshaw.io/sqlite v0.3.2 => ./third_party/sqlite
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999
git.sr.ht/~sbinet/gg v0.3.1
github.com/99designs/gqlgen v0.17.22
github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96
github.com/BurntSushi/toml v1.2.1
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802
github.com/CloudyKit/fastprinter v0.0.0-20200109182630-33d98a066a53
github.com/CloudyKit/jet/v6 v6.1.0
github.com/Jorropo/jsync v1.0.1
github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible
github.com/NebulousLabs/fastrand v0.0.0-20181203155948-6fb6489aac4e
github.com/NebulousLabs/go-upnp v0.0.0-20180202185039-29b680b06c82
github.com/OneOfOne/xxhash v1.2.2
github.com/Shopify/goreferrer v0.0.0-20220729165902-8cddb4f5de06
github.com/Shopify/sarama v1.19.0
github.com/Shopify/toxiproxy v2.1.4+incompatible
github.com/VividCortex/gohistogram v1.0.0
github.com/Yawning/aez v0.0.0-20211027044916-e49e68abd344
github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da
github.com/aead/siphash v1.0.1
github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5
github.com/agnivade/levenshtein v1.1.1
github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b
github.com/alecthomas/kingpin/v2 v2.3.2
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883
github.com/andybalholm/brotli v1.0.4
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
github.com/antihax/optional v1.0.0
github.com/apache/thrift v0.13.0
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da
github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310
github.com/aryann/difflib v0.0.0-20170710044230-e206f873d14a
github.com/aws/aws-lambda-go v1.13.3
github.com/aws/aws-sdk-go v1.27.0
github.com/aws/aws-sdk-go-v2 v0.18.0
github.com/aymerick/douceur v0.2.0
github.com/benbjohnson/clock v1.3.5
github.com/beorn7/perks v1.0.1
github.com/bernerdschaefer/eventsource v0.0.0-20130606115634-220e99a79763
github.com/bgentry/speakeasy v0.1.0
github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
github.com/btcsuite/btcd v0.23.3
github.com/btcsuite/btcd/btcec/v2 v2.2.1
github.com/btcsuite/btcd/btcutil v1.1.2
github.com/btcsuite/btcd/btcutil/psbt v1.1.5
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.1
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d
github.com/btcsuite/btcwallet v0.15.1
github.com/btcsuite/btcwallet/wallet/txauthor v1.2.3
github.com/btcsuite/btcwallet/wallet/txrules v1.2.0
github.com/btcsuite/btcwallet/wallet/txsizes v1.1.0
github.com/btcsuite/btcwallet/walletdb v1.4.0
github.com/btcsuite/btcwallet/wtxmgr v1.5.0
github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd
github.com/btcsuite/goleveldb v1.0.0
github.com/btcsuite/snappy-go v1.0.0
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792
github.com/btcsuite/winsvc v1.0.0
github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23
github.com/burdiyan/go/mainutil v0.0.0-20200124222818-6f87e0e684b6
github.com/casbin/casbin/v2 v2.1.2
github.com/cenkalti/backoff v2.2.1+incompatible
github.com/cenkalti/backoff/v4 v4.2.1
github.com/census-instrumentation/opencensus-proto v0.4.1
github.com/certifi/gocertifi v0.0.0-20200922220541-2c3bb06c6054
github.com/cespare/xxhash v1.1.0
github.com/cespare/xxhash/v2 v2.2.0
github.com/chromedp/cdproto v0.0.0-20230802225258-3cf4e6d46a89
github.com/chromedp/chromedp v0.9.2
github.com/chromedp/sysutil v1.0.0
github.com/chzyer/readline v1.5.1
github.com/cilium/ebpf v0.9.1
github.com/clbanning/x2j v0.0.0-20191024224557-825249438eec
github.com/client9/misspell v0.3.4
github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe
github.com/cncf/xds/go v0.0.0-20230105202645-06c439db220b
github.com/cockroachdb/datadriven v0.0.0-20200714090401-bf6692d28da5
github.com/cockroachdb/errors v1.2.4
github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd
github.com/codegangsta/inject v0.0.0-20150114235600-33e0aa1cb7c0
github.com/containerd/cgroups v1.1.0
github.com/coreos/bbolt v1.3.2
github.com/coreos/etcd v3.3.13+incompatible
github.com/coreos/go-semver v0.3.0
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f
github.com/coreos/go-systemd/v22 v22.5.0
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f
github.com/cpuguy83/go-md2man/v2 v2.0.2
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3
github.com/creack/pty v1.1.11
github.com/cskr/pubsub v1.0.2
github.com/davecgh/go-spew v1.1.1
github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c
github.com/decred/dcrd/crypto/blake256 v1.0.1
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0
github.com/decred/dcrd/lru v1.1.1
github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f
github.com/dgraph-io/badger v1.6.2
github.com/dgraph-io/ristretto v0.0.2
github.com/dgrijalva/jwt-go v3.2.0+incompatible
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954
github.com/dgryski/trifles v0.0.0-20200323201526-dd97f9abfb48
github.com/docker/go-units v0.5.0
github.com/dsnet/compress v0.0.1
github.com/dustin/go-humanize v1.0.0
github.com/dvyukov/go-fuzz v0.0.0-20210602112143-b1f3d6f4ef4e
github.com/eapache/go-resiliency v1.1.0
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21
github.com/eapache/queue v1.1.0
github.com/edsrzf/mmap-go v1.0.0
github.com/eknkc/amber v0.0.0-20171010120322-cdade1c07385
github.com/elastic/gosigar v0.14.2
github.com/envoyproxy/go-control-plane v0.10.3
github.com/envoyproxy/protoc-gen-validate v0.9.1
github.com/fatih/color v1.7.0
github.com/fatih/structs v1.1.0
github.com/fergusstrange/embedded-postgres v1.10.0
github.com/flosch/pongo2/v4 v4.0.2
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
github.com/flynn/noise v1.0.0
github.com/form3tech-oss/jwt-go v3.2.3+incompatible
github.com/francoispqt/gojay v1.2.13
github.com/franela/goblin v0.0.0-20200105215937-c9ffbefa60db
github.com/franela/goreq v0.0.0-20171204163338-bcd34c9993f8
github.com/frankban/quicktest v1.14.6
github.com/fsnotify/fsnotify v1.5.4
github.com/fxamacker/cbor/v2 v2.4.0
github.com/gabriel-vasile/mimetype v1.4.1
github.com/getsentry/raven-go v0.2.0
github.com/getsentry/sentry-go v0.16.0
github.com/ghodss/yaml v1.0.0
github.com/gin-contrib/sse v0.1.0
github.com/gin-gonic/gin v1.8.1
github.com/gliderlabs/ssh v0.1.1
github.com/go-errors/errors v1.4.2
github.com/go-fonts/liberation v0.3.0
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4
github.com/go-kit/kit v0.10.0
github.com/go-kit/log v0.2.1
github.com/go-latex/latex v0.0.0-20230307184459-12ec69307ad9
github.com/go-logfmt/logfmt v0.5.1
github.com/go-logr/logr v1.2.4
github.com/go-logr/stdr v1.2.2
github.com/go-martini/martini v0.0.0-20170121215854-22fa46961aab
github.com/go-pdf/fpdf v0.6.0
github.com/go-playground/assert/v2 v2.0.1
github.com/go-playground/locales v0.14.0
github.com/go-playground/universal-translator v0.18.0
github.com/go-playground/validator/v10 v10.11.1
github.com/go-sql-driver/mysql v1.4.0
github.com/go-stack/stack v1.8.0
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572
github.com/go-yaml/yaml v2.1.0+incompatible
github.com/gobwas/httphead v0.1.0
github.com/gobwas/pool v0.2.1
github.com/gobwas/ws v1.2.1
github.com/goccy/go-json v0.9.11
github.com/godbus/dbus/v5 v5.1.0
github.com/gogo/googleapis v1.1.0
github.com/gogo/protobuf v1.3.2
github.com/golang-jwt/jwt v3.2.2+incompatible
github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0
github.com/golang/glog v1.0.0
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7
github.com/golang/mock v1.6.0
github.com/golang/protobuf v1.5.3
github.com/golang/snappy v0.0.4
github.com/google/btree v1.0.1
github.com/google/go-cmp v0.5.9
github.com/google/go-github v17.0.0+incompatible
github.com/google/go-querystring v1.0.0
github.com/google/gofuzz v1.0.0
github.com/google/gopacket v1.1.19
github.com/google/martian v2.1.0+incompatible
github.com/google/pprof v0.0.0-20231023181126-ff6d637d2a7b
github.com/google/renameio v0.1.0
github.com/google/uuid v1.3.0
github.com/googleapis/gax-go v2.0.0+incompatible
github.com/googleapis/gax-go/v2 v2.0.5
github.com/gopherjs/gopherjs v0.0.0-20190430165422-3e4dfb77656c
github.com/gorilla/context v1.1.1
github.com/gorilla/css v1.0.0
github.com/gorilla/mux v1.8.0
github.com/gorilla/websocket v1.5.0
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/grpc-ecosystem/grpc-gateway v1.16.0
github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0
github.com/gxed/hashland/keccakpg v0.0.1
github.com/gxed/hashland/murmur3 v0.0.1
github.com/hashicorp/consul/api v1.3.0
github.com/hashicorp/consul/sdk v0.3.0
github.com/hashicorp/errwrap v1.1.0
github.com/hashicorp/go-cleanhttp v0.5.1
github.com/hashicorp/go-immutable-radix v1.0.0
github.com/hashicorp/go-msgpack v0.5.3
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/go-rootcerts v1.0.0
github.com/hashicorp/go-sockaddr v1.0.0
github.com/hashicorp/go-syslog v1.0.0
github.com/hashicorp/go-uuid v1.0.1
github.com/hashicorp/go-version v1.2.0
github.com/hashicorp/go.net v0.0.1
github.com/hashicorp/golang-lru v0.5.4
github.com/hashicorp/golang-lru/arc/v2 v2.0.5
github.com/hashicorp/golang-lru/v2 v2.0.5
github.com/hashicorp/hcl v1.0.0
github.com/hashicorp/logutils v1.0.0
github.com/hashicorp/mdns v1.0.0
github.com/hashicorp/memberlist v0.1.3
github.com/hashicorp/serf v0.8.2
github.com/hpcloud/tail v1.0.0
github.com/hudl/fargo v1.3.0
github.com/huin/goupnp v1.3.0
github.com/ianlancetaylor/demangle v0.0.0-20230524184225-eabc099b10ab
github.com/improbable-eng/grpc-web v0.15.0
github.com/inconshreveable/mousetrap v1.0.0
github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d
github.com/ipfs/bbloom v0.0.4
github.com/ipfs/boxo v0.15.0
github.com/ipfs/go-bitfield v1.1.0
github.com/ipfs/go-block-format v0.1.2
github.com/ipfs/go-cid v0.4.1
github.com/ipfs/go-cidutil v0.1.0
github.com/ipfs/go-datastore v0.6.0
github.com/ipfs/go-detect-race v0.0.1
github.com/ipfs/go-ds-badger v0.3.0
github.com/ipfs/go-ds-leveldb v0.5.0
github.com/ipfs/go-ipfs-blocksutil v0.0.1
github.com/ipfs/go-ipfs-delay v0.0.1
github.com/ipfs/go-ipfs-pq v0.0.3
github.com/ipfs/go-ipfs-redirects-file v0.1.1
github.com/ipfs/go-ipfs-util v0.0.3
github.com/ipfs/go-ipld-cbor v0.0.6
github.com/ipfs/go-ipld-format v0.5.0
github.com/ipfs/go-ipld-legacy v0.2.1
github.com/ipfs/go-log v1.0.5
github.com/ipfs/go-log/v2 v2.5.1
github.com/ipfs/go-metrics-interface v0.0.1
github.com/ipfs/go-peertaskqueue v0.8.1
github.com/ipfs/go-unixfs v0.4.5
github.com/ipfs/go-unixfsnode v1.7.1
github.com/ipld/go-car/v2 v2.10.2-0.20230622090957-499d0c909d33
github.com/ipld/go-codec-dagpb v1.6.0
github.com/ipld/go-ipld-prime v0.21.0
github.com/iris-contrib/jade v1.1.4
github.com/iris-contrib/schema v0.0.6
github.com/jackc/chunkreader/v2 v2.0.1
github.com/jackc/pgconn v1.10.0
github.com/jackc/pgio v1.0.0
github.com/jackc/pgpassfile v1.0.0
github.com/jackc/pgproto3/v2 v2.1.1
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b
github.com/jackc/pgtype v1.8.1
github.com/jackc/pgx/v4 v4.13.0
github.com/jackc/puddle v1.1.3
github.com/jackpal/gateway v1.0.5
github.com/jackpal/go-nat-pmp v1.0.2
github.com/jbenet/go-cienv v0.1.0
github.com/jbenet/go-temp-err-catcher v0.1.0
github.com/jbenet/goprocess v0.1.4
github.com/jedib0t/go-pretty/v6 v6.2.7
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
github.com/jessevdk/go-flags v1.4.0
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
github.com/jonboulle/clockwork v0.2.2
github.com/josharian/intern v1.0.0
github.com/jpillora/backoff v1.0.0
github.com/jrick/logrotate v1.0.0
github.com/json-iterator/go v1.1.12
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024
github.com/jtolds/gls v4.20.0+incompatible
github.com/juju/loggo v0.0.0-20210728185423-eebad3a902c4
github.com/juju/testing v0.0.0-20220203020004-a0ff61f03494
github.com/julienschmidt/httprouter v1.3.0
github.com/kataras/blocks v0.0.7
github.com/kataras/golog v0.1.7
github.com/kataras/iris/v12 v12.2.0-beta5
github.com/kataras/pio v0.0.11
github.com/kataras/sitemap v0.0.6
github.com/kataras/tunnel v0.0.4
github.com/kevinmbeaulieu/eq-go v1.0.0
github.com/kisielk/errcheck v1.5.0
github.com/kisielk/gotool v1.0.0
github.com/kkdai/bstream v1.0.0
github.com/klauspost/compress v1.17.2
github.com/klauspost/cpuid/v2 v2.2.5
github.com/klauspost/pgzip v1.2.5
github.com/konsorten/go-windows-terminal-sequences v1.0.3
github.com/koron/go-ssdp v0.0.4
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
github.com/kr/pretty v0.3.1
github.com/kr/pty v1.1.3
github.com/kr/text v0.2.0
github.com/labstack/echo/v4 v4.9.0
github.com/labstack/gommon v0.3.1
github.com/leodido/go-urn v1.2.1
github.com/lib/pq v1.10.7
github.com/libp2p/go-buffer-pool v0.1.0
github.com/libp2p/go-cidranger v1.1.0
github.com/libp2p/go-doh-resolver v0.4.0
github.com/libp2p/go-flow-metrics v0.1.0
github.com/libp2p/go-libp2p v0.32.1
github.com/libp2p/go-libp2p-asn-util v0.3.0
github.com/libp2p/go-libp2p-gostream v0.6.0
github.com/libp2p/go-libp2p-kad-dht v0.25.1
github.com/libp2p/go-libp2p-kbucket v0.6.3
github.com/libp2p/go-libp2p-record v0.2.0
github.com/libp2p/go-libp2p-routing-helpers v0.7.2
github.com/libp2p/go-libp2p-testing v0.12.0
github.com/libp2p/go-libp2p-xor v0.1.0
github.com/libp2p/go-msgio v0.3.0
github.com/libp2p/go-nat v0.2.0
github.com/libp2p/go-netroute v0.2.1
github.com/libp2p/go-openssl v0.1.0
github.com/libp2p/go-reuseport v0.4.0
github.com/libp2p/go-yamux/v4 v4.0.1
github.com/libp2p/zeroconf/v2 v2.2.0
github.com/lightninglabs/gozmq v0.0.0-20191113021534-d20a764486bf
github.com/lightninglabs/neutrino v0.14.2
github.com/lightninglabs/protobuf-hex-display v1.4.3-hex-display
github.com/lightningnetwork/lightning-onion v1.0.2-0.20220211021909-bb84a1ccb0c5
github.com/lightningnetwork/lnd v0.15.1-beta.rc2
github.com/lightningnetwork/lnd/cert v1.1.1
github.com/lightningnetwork/lnd/clock v1.1.0
github.com/lightningnetwork/lnd/healthcheck v1.2.2
github.com/lightningnetwork/lnd/kvdb v1.3.1
github.com/lightningnetwork/lnd/queue v1.1.0
github.com/lightningnetwork/lnd/ticker v1.1.0
github.com/lightningnetwork/lnd/tlv v1.0.3
github.com/lightningnetwork/lnd/tor v1.1.0
github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-bc2310a04743
github.com/lightstep/lightstep-tracer-go v0.18.1
github.com/logrusorgru/aurora/v3 v3.0.0
github.com/ltcsuite/ltcd v0.0.0-20190101042124-f37f8bf35796
github.com/lunixbochs/vtclean v1.0.0
github.com/lyft/protoc-gen-validate v0.0.13
github.com/magiconair/properties v1.8.1
github.com/mailgun/raymond/v2 v2.0.46
github.com/mailru/easyjson v0.7.7
github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd
github.com/matryer/moq v0.2.7
github.com/mattn/go-colorable v0.1.13
github.com/mattn/go-isatty v0.0.20
github.com/mattn/go-pointer v0.0.1
github.com/mattn/go-runewidth v0.0.13
github.com/matttproud/golang_protobuf_extensions v1.0.4
github.com/mholt/archiver/v3 v3.5.0
github.com/microcosm-cc/bluemonday v1.0.21
github.com/miekg/dns v1.1.56
github.com/mikioh/tcp v0.0.0-20190314235350-803a9b46060c
github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b
github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1
github.com/minio/sha256-simd v1.0.1
github.com/mitchellh/cli v1.0.0
github.com/mitchellh/go-homedir v1.1.0
github.com/mitchellh/go-testing-interface v1.0.0
github.com/mitchellh/gox v0.4.0
github.com/mitchellh/iochan v1.0.0
github.com/mitchellh/mapstructure v1.5.0
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
github.com/modern-go/reflect2 v1.0.2
github.com/mr-tron/base58 v1.2.0
github.com/multiformats/go-base32 v0.1.0
github.com/multiformats/go-base36 v0.2.0
github.com/multiformats/go-multiaddr v0.12.0
github.com/multiformats/go-multiaddr-dns v0.3.1
github.com/multiformats/go-multiaddr-fmt v0.1.0
github.com/multiformats/go-multibase v0.2.0
github.com/multiformats/go-multicodec v0.9.0
github.com/multiformats/go-multihash v0.2.3
github.com/multiformats/go-multistream v0.5.0
github.com/multiformats/go-varint v0.0.7
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
github.com/mwitkow/grpc-proxy v0.0.0-20181017164139-0f1106ef9c76
github.com/nats-io/jwt v0.3.2
github.com/nats-io/nats-server/v2 v2.1.2
github.com/nats-io/nats.go v1.9.1
github.com/nats-io/nkeys v0.1.3
github.com/nats-io/nuid v1.0.1
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab
github.com/nwaples/rardecode v1.1.2
github.com/nxadm/tail v1.4.4
github.com/oklog/oklog v0.3.2
github.com/oklog/run v1.1.0
github.com/oklog/ulid v1.3.1
github.com/oklog/ulid/v2 v2.1.0
github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5
github.com/onsi/ginkgo v1.14.0
github.com/onsi/ginkgo/v2 v2.13.0
github.com/onsi/gomega v1.27.10
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
github.com/opencontainers/runtime-spec v1.1.0
github.com/opentracing-contrib/go-observer v0.0.0-20170622124052-a52f23424492
github.com/opentracing/basictracer-go v1.0.0
github.com/opentracing/opentracing-go v1.2.0
github.com/openzipkin-contrib/zipkin-go-opentracing v0.4.5
github.com/openzipkin/zipkin-go v0.4.1
github.com/pact-foundation/pact-go v1.0.4
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58
github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30
github.com/pborman/uuid v1.2.0
github.com/pelletier/go-toml v1.9.5
github.com/pelletier/go-toml/v2 v2.0.5
github.com/performancecopilot/speed v3.0.0+incompatible
github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9
github.com/peterbourgon/ff/v3 v3.3.0
github.com/peterbourgon/ff/v4 v4.0.0-alpha.3
github.com/peterbourgon/trc v0.0.3
github.com/pierrec/lz4 v2.0.5+incompatible
github.com/pierrec/lz4/v4 v4.1.8
github.com/pingcap/errors v0.11.4
github.com/pion/datachannel v1.5.5
github.com/pion/dtls/v2 v2.2.7
github.com/pion/ice/v2 v2.3.6
github.com/pion/interceptor v0.1.17
github.com/pion/logging v0.2.2
github.com/pion/mdns v0.0.7
github.com/pion/randutil v0.1.0
github.com/pion/rtcp v1.2.10
github.com/pion/rtp v1.7.13
github.com/pion/sctp v1.8.7
github.com/pion/sdp/v3 v3.0.6
github.com/pion/srtp/v2 v2.0.15
github.com/pion/stun v0.6.0
github.com/pion/transport/v2 v2.2.1
github.com/pion/turn/v2 v2.1.0
github.com/pion/webrtc/v3 v3.2.9
github.com/pkg/errors v0.9.1
github.com/pkg/profile v1.2.1
github.com/planetscale/vtprotobuf v0.3.0
github.com/pmezard/go-difflib v1.0.0
github.com/polydawn/refmt v0.89.0
github.com/posener/complete v1.1.1
github.com/prometheus/client_golang v1.16.0
github.com/prometheus/client_model v0.4.0
github.com/prometheus/common v0.44.0
github.com/prometheus/procfs v0.11.1
github.com/prometheus/tsdb v0.7.1
github.com/quic-go/qpack v0.4.0
github.com/quic-go/qtls-go1-19 v0.2.1
github.com/quic-go/qtls-go1-20 v0.3.4
github.com/quic-go/quic-go v0.39.3
github.com/quic-go/webtransport-go v0.6.0
github.com/raulk/go-watchdog v1.3.0
github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a
github.com/rivo/uniseg v0.2.0
github.com/rogpeppe/fastuuid v1.2.0
github.com/rogpeppe/go-internal v1.10.0
github.com/rs/cors v1.7.0
github.com/russross/blackfriday v1.5.2
github.com/russross/blackfriday/v2 v2.1.0
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f
github.com/samber/lo v1.36.0
github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da
github.com/sanity-io/litter v1.5.5
github.com/schollz/closestmatch v2.1.0+incompatible
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529
github.com/sergi/go-diff v1.2.0
github.com/sethvargo/go-retry v0.2.4
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82
github.com/shurcooL/sanitized_anchor_name v1.0.0
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133
github.com/sirupsen/logrus v1.9.0
github.com/smartystreets/assertions v1.2.0
github.com/smartystreets/goconvey v1.7.2
github.com/soheilhy/cmux v0.1.5
github.com/sony/gobreaker v0.4.1
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572
github.com/spaolacci/murmur3 v1.1.0
github.com/spf13/afero v1.1.2
github.com/spf13/cast v1.3.0
github.com/spf13/cobra v1.1.3
github.com/spf13/jwalterweatherman v1.0.0
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.7.0
github.com/streadway/amqp v0.0.0-20190827072141-edfb9018d271
github.com/streadway/handy v0.0.0-20190108123426-d5acb3125c2a
github.com/stretchr/objx v0.5.0
github.com/stretchr/testify v1.8.4
github.com/subosito/gotenv v1.2.0
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07
github.com/tdewolff/minify/v2 v2.12.4
github.com/tdewolff/parse/v2 v2.6.4
github.com/tidwall/btree v1.6.0
github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802
github.com/tv42/zbase32 v0.0.0-20160707012821-501572607d02
github.com/tyler-smith/go-bip39 v1.1.0
github.com/ucarion/urlpath v0.0.0-20200424170820-7ccc79b76bbb
github.com/ugorji/go v1.1.7
github.com/ugorji/go/codec v1.2.7
github.com/ulikunitz/xz v0.5.10
github.com/urfave/cli v1.22.10
github.com/urfave/cli/v2 v2.23.7
github.com/urfave/negroni v1.0.0
github.com/valyala/bytebufferpool v1.0.0
github.com/valyala/fasthttp v1.40.0
github.com/valyala/fasttemplate v1.2.1
github.com/vektah/gqlparser/v2 v2.5.1
github.com/viant/assertly v0.4.8
github.com/viant/toolbox v0.24.0
github.com/vmihailenco/msgpack/v5 v5.3.5
github.com/vmihailenco/tagparser/v2 v2.0.0
github.com/warpfork/go-testmark v0.12.1
github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0
github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc
github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11
github.com/whyrusleeping/cbor-gen v0.0.0-20230418232409-daab9ece03a0
github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f
github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1
github.com/x448/float16 v0.8.4
github.com/xhit/go-str2duration/v2 v2.1.0
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673
github.com/yosssi/ace v0.0.5
github.com/yuin/goldmark v1.4.13
gitlab.com/yawning/bsaes.git v0.0.0-20190805113838-0a714cd429ec
go.etcd.io/bbolt v1.3.6
go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738
go.etcd.io/etcd/api/v3 v3.5.5
go.etcd.io/etcd/client/pkg/v3 v3.5.5
go.etcd.io/etcd/client/v2 v2.305.5
go.etcd.io/etcd/client/v3 v3.5.5
go.etcd.io/etcd/pkg/v3 v3.5.5
go.etcd.io/etcd/raft/v3 v3.5.5
go.etcd.io/etcd/server/v3 v3.5.5
go.opencensus.io v0.24.0
go.opentelemetry.io/contrib v0.20.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.25.0
go.opentelemetry.io/otel v1.16.0
go.opentelemetry.io/otel/exporters/jaeger v1.14.0
go.opentelemetry.io/otel/exporters/otlp v0.20.0
go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.14.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.14.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.14.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.14.0
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.14.0
go.opentelemetry.io/otel/exporters/zipkin v1.14.0
go.opentelemetry.io/otel/metric v1.16.0
go.opentelemetry.io/otel/sdk v1.16.0
go.opentelemetry.io/otel/sdk/export/metric v0.20.0
go.opentelemetry.io/otel/sdk/metric v0.20.0
go.opentelemetry.io/otel/trace v1.16.0
go.opentelemetry.io/proto/otlp v0.19.0
go.uber.org/atomic v1.11.0
go.uber.org/dig v1.17.1
go.uber.org/fx v1.20.1
go.uber.org/goleak v1.2.0
go.uber.org/mock v0.3.0
go.uber.org/multierr v1.11.0
go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee
go.uber.org/zap v1.26.0
go4.org v0.0.0-20180809161055-417644f6feb5
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d
golang.org/x/crypto v0.14.0
golang.org/x/exp v0.0.0-20231006140011-7918f672742d
golang.org/x/image v0.6.0
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028
golang.org/x/mod v0.13.0
golang.org/x/net v0.17.0
golang.org/x/oauth2 v0.8.0
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
golang.org/x/sync v0.4.0
golang.org/x/sys v0.13.0
golang.org/x/term v0.13.0
golang.org/x/text v0.13.0
golang.org/x/time v0.0.0-20220922220347-f3bd1da661af
golang.org/x/tools v0.14.0
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2
gonum.org/v1/gonum v0.13.0
gonum.org/v1/plot v0.10.1
google.golang.org/api v0.13.0
google.golang.org/appengine v1.6.7
google.golang.org/genproto v0.0.0-20230110181048-76db0878b65f
google.golang.org/grpc v1.53.0
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.2.0
google.golang.org/protobuf v1.31.0
gopkg.in/alecthomas/kingpin.v2 v2.2.6
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
gopkg.in/cheggaaa/pb.v1 v1.0.25
gopkg.in/errgo.v1 v1.0.1
gopkg.in/errgo.v2 v2.1.0
gopkg.in/fsnotify.v1 v1.4.7
gopkg.in/gcfg.v1 v1.2.3
gopkg.in/inf.v0 v0.9.1
gopkg.in/ini.v1 v1.67.0
gopkg.in/macaroon-bakery.v2 v2.0.1
gopkg.in/macaroon.v2 v2.0.0
gopkg.in/natefinch/lumberjack.v2 v2.0.0
gopkg.in/resty.v1 v1.12.0
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
gopkg.in/warnings.v0 v0.1.2
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
honnef.co/go/tools v0.0.1-2019.2.3
lukechampine.com/blake3 v1.2.1
nhooyr.io/websocket v1.8.7
rsc.io/binaryregexp v0.2.0
rsc.io/pdf v0.1.1
sigs.k8s.io/yaml v1.2.0
sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0
sourcegraph.com/sourcegraph/go-diff v0.5.0
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4
marten-seemann commented 1 year ago

It sounds like there are 2 issues here:

  1. Peers having no protocols.
  2. NewStream failing.

Regarding (1), I suspect that this is due to the cleanup logic, although I can't see any bug. When a peer disconnects (i.e. when all of the (potentially multiple) connection we had with a peer are gone), we clean up the protocol list from the peer store after a short period. You can find the logic here: https://github.com/libp2p/go-libp2p/blob/fc1c9f5d4e7aef936847ffa8915a8eaffb34eba2/p2p/host/pstoremanager/pstoremanager.go#L81.

The bug here is pretty obvious, we call RemovePeer even if the peer has connected back to us in the mean time. This will leave the peer list of addresses permanently empty, at least until the next Identify event happens (which doesn't happen frequently in practice, depending on your application).

Regarding (2), you're probably running into this error: https://github.com/libp2p/go-libp2p/blob/fc1c9f5d4e7aef936847ffa8915a8eaffb34eba2/p2p/host/basic/basic_host.go#L696 I'm not sure why that happens, the peer should have completed the multistream protocol negotiation.

marten-seemann commented 1 year ago

The bug here is pretty obvious, we call RemovePeer even if the peer has connected back to us in the mean time. This will leave the peer list of addresses permanently empty, at least until the next Identify event happens (which doesn't happen frequently in practice, depending on your application).

I'm not sure if this is true. We clean up disconnected when the peer reconnects: https://github.com/libp2p/go-libp2p/blob/fc1c9f5d4e7aef936847ffa8915a8eaffb34eba2/p2p/host/pstoremanager/pstoremanager.go#L108-L111

I think there's still a race condition, since eventbus notifications are async. The peer might have reconnected, identify completed, the timer fired and RemovePeer called before we process the Connected event. We should fix this by actively checking if the peer is connected before calling RemovePeer.

This soulds like an unlikely event though, and I'm not sure if it explains what you're seeing.

marten-seemann commented 1 year ago

@juligasa How do you know that you had the list of protocols for the peer at one point, and that they then disappear? Do you have any logs?

juligasa commented 1 year ago

In our particular case we always check the protocols right after doing host.Connect() successfully, so That's why I think it does not have to do with cleaning protocols on disconnected peers. I have created a test branch with multiple traces to see whats going on: https://github.com/MintterHypermedia/mintter/blob/playground/zero-protocols/backend/mttnet/connect.go#L86-L124

marten-seemann commented 1 year ago

Do you make sure that this code is run after Identify completed?

This line doesn't look correct, you'll never be able to return a stream opened after the list was clear. https://github.com/MintterHypermedia/mintter/blob/7666ed6622e48abe494da77fb946e14f9cabe92f/backend/mttnet/connect.go#L98

juligasa commented 1 year ago

In production we have a retry loop (that eventually times out) that gets protocols every 50ms with the hope of identifying completes first, but no luck. The stream creation was for testing purposes to see if there was any hint.

aschmahmann commented 1 year ago

@marten-seemann I wonder if this looks similar to this problem https://discuss.ipfs.tech/t/files-are-rarely-available-through-public-gateways/17144 (which you thought might be resource manager related)

marten-seemann commented 1 year ago

In production we have a retry loop (that eventually times out) that gets protocols every 50ms with the hope of identifying completes first, but no luck.

Are you familiar with the event bus? You can subscribe to the identify event, which will fire once the list of protocols is available.

burdiyan commented 1 year ago

Are you familiar with the event bus? You can subscribe to the identify event, which will fire once the list of protocols is available.

@marten-seemann I guess that wouldn't be necessary, because libp2p currently waits for Identify to complete during Connect, so we assume that checking for protocols after Connect successfully returns should work, but sometimes it doesn't.

https://github.com/libp2p/go-libp2p/blob/5c9583419be0c777d13c931e0003ea69c4b90dad/p2p/host/basic/basic_host.go#L751

So, looks like the problem is that sometimes we get to have peers that we've connected with, and we've done the identify with, but even after that we have 0 protocols for them in our peer store.

juligasa commented 1 year ago

Are you familiar with the event bus? You can subscribe to the identify event, which will fire once the list of protocols is available.

I see, we could do that. However, I wonder why the response from some peers is instant, and for other peers it takes +10s¿? we give up retrying at 10 seconds which is beyond the user's expectation to connect to a peer that is apparently online. After restarting the app, we get a new random set of peers reporting empty protocols.

burdiyan commented 1 year ago

To summarize, the problem we're facing at Mintter is:

  1. libp2p.Connect successfully returns. Which assuming the current code in libp2p also means that Identify would have already happened.
  2. Regardless, for some apparently random subset of peers we wouldn't have the list of supported protocols in the peer store.
  3. In addition to that, NewStream would also fail for those peers.

I was hoping that libp2p 1.32 would have fixed the issue (there's a mention of fixing the error of not waiting for transient connections, which sounded like exactly the problem we face), but according to the tests @juligasa made recently, the problem is still happening.

sukunrt commented 1 year ago

I'm confused how we can get Context Deadline exceeded(from NewStream) after the peer reported zero protocols. In MintterHypermedia/mintter@playground/zero-protocols/backend/mttnet/connect.go#L86-L124, I see that the error on NewStream is not wrapped or printed anywhere. Can you print err too here. Also, is it possible to share log lines for a zero protocols peer?

@aschmahmann that's unlikely because in that case the err is supposed to be stream reset or protocol negotiation failed. We shouldn't see Context Deadline Exceeded with that issues.

juligasa commented 1 year ago

Sure. I re-run the whole test again. But the results are a bit different from the last time. Now I get fewer zero-protocol errors, and most of them are at the beginning. At the next sync round (we do one every minute) they seem to have protocols. However, for those fewer cases where the protocol is still failing, we get three type of outcomes.

  1. Either we can get a NewStream
  2. Failing a NewStream with the error context deadline exceeded
  3. Failing a NewStream with the error failed to negotiate protocol: context deadline exceeded

The difference at the network level with the tests in weeks before, is that by now, most of the network should have upgraded to the latest code which includes the libp2p upgrade from v0.31.0 to v0.32.1

sukunrt commented 11 months ago

Apologies for the delay here.

I don't have the complete explanation of what's causing this, but my best guess currently is that the peer has gone away and is unresponsive.

Can you run your tests with GOLOG_LOG_LEVEL="net/identify=debug" env flag set and is it possible to increase this context timeout to 45 seconds(30 seconds for the connection keep alive to expire and 15 as a buffer). Within 45 seconds if the peer is indeed unresponsive we should close the connection and the error will now change to ErrStreamReset. Checking identify debug logs will tell us why the initial identify failed, leading to 0 protocols.

juligasa commented 11 months ago

I ran the test again. This time the difference is that after zero protocols I always can open a stream with the peer reporting zero protocols. So those peers shouldn't have gone away. The timeout does not make any difference here As for the net/identify=debug, is super verbose, don't really know what to look at.

sukunrt commented 11 months ago

As I understand, now you can open the streams so the only problem is that these peers have provided zero protocols in identify or potentially identify has failed.

In the identify logs can you check if there are any entries for the peer that has zero protocols?

sukunrt commented 11 months ago

In particular, I'm interested in this line: https://github.com/libp2p/go-libp2p/blob/master/p2p/protocol/identify/id.go#L399C2-L399C2

juligasa commented 11 months ago

As I understand, now you can open the streams so the only problem is that these peers have provided zero protocols in identify or potentially identify has failed.

Correct As for the failed to identify traces. I see them in two flavors:

  1. 2023-12-22T16:46:32.485+0100 WARN net/identify identify/id.go:399 failed to identify 12D3KooWHZG1TAfuW2ARQm3Anq8QVhqBwWgnyZpn8U5xuxz1qA5v: Application error 0x1 (remote): conn-5798258: system: cannot reserve inbound connection: resource limit exceeded 95% of the occurences
  2. 2023-12-22T16:46:32.575+0100 WARN net/identify identify/id.go:399 failed to identify 12D3KooWEhAvHBjfy1at22dJisN8wvD8KvvCyNYh45smcsMFrLML: Application error 0x47415445 (remote): connection gated 5% of the occurences

But none of the peers on those traces correspong with any peer reporting 0 protocols. Actually, searching for peers reporting zero protocols, have quite sensible traces, like

2023-12-22T16:46:33.852+0100    DEBUG   net/identify    identify/id.go:512      /ipfs/id/1.0.0 received message from 12D3KooWJmT4B5QVfpBXh1XMJyq1dV9L6e64mwWc4peSU9h49M6A /ip4/23.20.24.146/udp/4002/quic-v1/p2p/12D3KooWNmjM4sMbSkDEA6ShvjTgkrJHjMya46fhZ9PjKZ4KVZYq/p2p-circuit
2023-12-22T16:46:33.852+0100    DEBUG   net/identify    identify/id.go:461      /ipfs/id/1.0.0 sending message to 12D3KooWJmT4B5QVfpBXh1XMJyq1dV9L6e64mwWc4peSU9h49M6A /ip4/23.20.24.146/udp/4002/quic-v1/p2p/12D3KooWNmjM4sMbSkDEA6ShvjTgkrJHjMya46fhZ9PjKZ4KVZYq/p2p-circuit
2023-12-22T16:46:33.852+0100    DEBUG   net/identify    identify/id.go:778      12D3KooWM38pA4drTk7zVttN6rgMVZyGdQfemj8sdLvt8QD6QZ8s received listen addrs for 12D3KooWJmT4B5QVfpBXh1XMJyq1dV9L6e64mwWc4peSU9h49M6A: [/ip4/23.20.24.146/tcp/4002/p2p/12D3KooWNmjM4sMbSkDEA6ShvjTgkrJHjMya46fhZ9PjKZ4KVZYq/p2p-circuit /ip4/23.20.24.146/udp/4002/quic-v1/p2p/12D3KooWNmjM4sMbSkDEA6ShvjTgkrJHjMya46fhZ9PjKZ4KVZYq/p2p-circuit /ip4/23.20.24.146/udp/4002/quic-v1/webtransport/certhash/uEiCKDdWcj0Ana-uOxOvfxPeCa5AySnFnndOF3IluqDCS8g/certhash/uEiDX-M06wPhYAwfCBWFYdGeJzF1WEicNV0t4D03wSc6tUA/p2p/12D3KooWNmjM4sMbSkDEA6ShvjTgkrJHjMya46fhZ9PjKZ4KVZYq/p2p-circuit /ip4/52.22.139.174/tcp/4002/p2p/12D3KooWGvsbBfcbnkecNoRBM7eUTiuriDqUyzu87pobZXSdUUsJ/p2p-circuit /ip4/52.22.139.174/udp/4002/quic-v1/p2p/12D3KooWGvsbBfcbnkecNoRBM7eUTiuriDqUyzu87pobZXSdUUsJ/p2p-circuit /ip4/52.22.139.174/udp/4002/quic-v1/webtransport/certhash/uEiDLL3ZwtJrzbvsBVY1qEZlnqdZzrZkuqoBerkADK7k7Tg/certhash/uEiBo3TlTcEq5bDfHELfzrR4fYGs4a7ja0Zm2313x5K8gpQ/p2p/12D3KooWGvsbBfcbnkecNoRBM7eUTiuriDqUyzu87pobZXSdUUsJ/p2p-circuit /ip4/192.168.1.78/tcp/56002 /ip4/192.168.1.78/udp/56002/quic-v1 /ip4/192.168.1.78/udp/56002/quic-v1/webtransport/certhash/uEiB25I72CuDBslKIys4RHN7-oUIA3DkGUt9cmp88ZD9DYQ/certhash/uEiCWnNHBBXctx6gu1Zc4atA3FZY3XMz_CKMVe7h-o6GzRw]
2023-12-22T16:46:34.363+0100    DEBUG   net/identify    identify/id.go:461      /ipfs/id/1.0.0 sending message to 12D3KooWJmT4B5QVfpBXh1XMJyq1dV9L6e64mwWc4peSU9h49M6A /ip4/88.1.95.131/udp/56002/quic-v1
2023-12-22T16:46:34.372+0100    DEBUG   net/identify    identify/id.go:512      /ipfs/id/1.0.0 received message from 12D3KooWJmT4B5QVfpBXh1XMJyq1dV9L6e64mwWc4peSU9h49M6A /ip4/88.1.95.131/udp/56002/quic-v1
2023-12-22T16:46:34.372+0100    DEBUG   net/identify    identify/id.go:778      12D3KooWM38pA4drTk7zVttN6rgMVZyGdQfemj8sdLvt8QD6QZ8s received listen addrs for 12D3KooWJmT4B5QVfpBXh1XMJyq1dV9L6e64mwWc4peSU9h49M6A: [/ip4/23.20.24.146/tcp/4002/p2p/12D3KooWNmjM4sMbSkDEA6ShvjTgkrJHjMya46fhZ9PjKZ4KVZYq/p2p-circuit /ip4/23.20.24.146/udp/4002/quic-v1/p2p/12D3KooWNmjM4sMbSkDEA6ShvjTgkrJHjMya46fhZ9PjKZ4KVZYq/p2p-circuit /ip4/23.20.24.146/udp/4002/quic-v1/webtransport/certhash/uEiCKDdWcj0Ana-uOxOvfxPeCa5AySnFnndOF3IluqDCS8g/certhash/uEiDX-M06wPhYAwfCBWFYdGeJzF1WEicNV0t4D03wSc6tUA/p2p/12D3KooWNmjM4sMbSkDEA6ShvjTgkrJHjMya46fhZ9PjKZ4KVZYq/p2p-circuit /ip4/52.22.139.174/tcp/4002/p2p/12D3KooWGvsbBfcbnkecNoRBM7eUTiuriDqUyzu87pobZXSdUUsJ/p2p-circuit /ip4/52.22.139.174/udp/4002/quic-v1/p2p/12D3KooWGvsbBfcbnkecNoRBM7eUTiuriDqUyzu87pobZXSdUUsJ/p2p-circuit /ip4/52.22.139.174/udp/4002/quic-v1/webtransport/certhash/uEiDLL3ZwtJrzbvsBVY1qEZlnqdZzrZkuqoBerkADK7k7Tg/certhash/uEiBo3TlTcEq5bDfHELfzrR4fYGs4a7ja0Zm2313x5K8gpQ/p2p/12D3KooWGvsbBfcbnkecNoRBM7eUTiuriDqUyzu87pobZXSdUUsJ/p2p-circuit /ip4/192.168.1.78/tcp/56002 /ip4/192.168.1.78/udp/56002/quic-v1 /ip4/192.168.1.78/udp/56002/quic-v1/webtransport/certhash/uEiB25I72CuDBslKIys4RHN7-oUIA3DkGUt9cmp88ZD9DYQ/certhash/uEiCWnNHBBXctx6gu1Zc4atA3FZY3XMz_CKMVe7h-o6GzRw]
2023-12-22T16:47:32.612+0100    DEBUG   net/identify    identify/id.go:461      /ipfs/id/1.0.0 sending message to 12D3KooWJmT4B5QVfpBXh1XMJyq1dV9L6e64mwWc4peSU9h49M6A /ip4/88.1.95.131/udp/56002/quic-v1
2023-12-22T16:47:32.667+0100    DEBUG   net/identify    identify/id.go:316      already sent this snapshot to peer      {"peer": "12D3KooWJmT4B5QVfpBXh1XMJyq1dV9L6e64mwWc4peSU9h49M6A", "seq": 7}

What about those resource limits errors? should be worried? related to this https://github.com/libp2p/go-libp2p/issues/2628 ?