search

libp2p / go-libp2p-kad-dht

A Kademlia DHT implementation on go-libp2p
https://github.com/libp2p/specs/tree/master/kad-dht
MIT License
530 stars 232 forks source link

The memory issue with using the package #735

Closed AlexiaChen closed 2 years ago

AlexiaChen commented 3 years ago

verison

go-kad-dht 0.12.2

Summary

First of all, thank you very much to the team of libp2p for such a great project.

Our project recently encountered a memory problem related to DHT, specifically: once the Discovery function is turned on, the memory will grow slowly, and after a few days of accumulation, the memory will reach the alert level of monitoring and eventually trigger an OOM.

The following is the data observed with pprof after several days, mainly using go-libp2p-swarm(*Conn)NewStream in DHT, which is called by the queryPeer method in DHT, from causing the memory to grow slowly.

Pprof data

memory data with pprof collected for a few days:

day 0:

svg-day-0.zip

day 1:

svg-day-1.zip

day 2:

svg-day-2.zip

Analysis

From the sampling of each time period above, basically most of the functions where svg memory takes the bulk of the time do not grow with time, some of them even go down instead, and none of these functions have memory leaks. Only the method go-yamux-newStream (called by go-libp2p-swarm(*Conn)NewStream) is growing by more than 1x each time.

Notice

After the above analysis, because it is related to Kad DHT queryPeer method, we try to adjust the Concurrency parameter from the default 10 to 1, (because concurrency affects the number of concurrent queryPeer) and finally there is no OOM, and the memory remains stable. I don't know why, it's a very strange phenomenon.

Stebalien commented 3 years ago

Could you run go list -m all in your project root? There was a memory leak in the TCP transport that has since been fixed. That may have caused the issue.

AlexiaChen commented 3 years ago

Could you run go list -m all in your project root? There was a memory leak in the TCP transport that has since been fixed. That may have caused the issue.

Hi @Stebalien, Thank you for your reply

here is log from go list -m all :

github.com/harmony-one/harmony
cloud.google.com/go v0.37.0
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999
github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9
github.com/Azure/azure-pipeline-go v0.2.2
github.com/Azure/azure-storage-blob-go v0.7.0
github.com/Azure/go-autorest/autorest v0.9.0
github.com/Azure/go-autorest/autorest/adal v0.8.0
github.com/Azure/go-autorest/autorest/date v0.2.0
github.com/Azure/go-autorest/autorest/mocks v0.3.0
github.com/Azure/go-autorest/logger v0.1.0
github.com/Azure/go-autorest/tracing v0.5.0
github.com/BurntSushi/toml v0.3.1
github.com/DataDog/zstd v1.4.5
github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible
github.com/Kubuxu/go-os-helper v0.0.1
github.com/OneOfOne/xxhash v1.2.5
github.com/OpenPeeDeeP/depguard v1.0.1
github.com/Shopify/sarama v1.19.0
github.com/Shopify/toxiproxy v2.1.4+incompatible
github.com/StackExchange/wmi v0.0.0-20180116203802-5d049714c4a6
github.com/VictoriaMetrics/fastcache v1.5.7
github.com/VividCortex/gohistogram v1.0.0
github.com/Workiva/go-datastructures v1.0.50
github.com/Zilliqa/gozilliqa-sdk v1.2.1-0.20201201074141-dd0ecada1be6
github.com/aead/siphash v1.0.1
github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
github.com/allegro/bigcache v1.2.1
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
github.com/apache/thrift v0.13.0
github.com/aristanetworks/goarista v0.0.0-20190607111240-52c2a7864a08
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da
github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310
github.com/aryann/difflib v0.0.0-20170710044230-e206f873d14a
github.com/aws/aws-lambda-go v1.13.3
github.com/aws/aws-sdk-go v1.30.1
github.com/aws/aws-sdk-go-v2 v0.18.0
github.com/beevik/ntp v0.3.0
github.com/benbjohnson/clock v1.0.3
github.com/beorn7/perks v1.0.1
github.com/bgentry/speakeasy v0.1.0
github.com/bombsimon/wsl v1.2.5
github.com/bombsimon/wsl/v2 v2.0.0
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
github.com/btcsuite/btcd v0.21.0-beta
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
github.com/btcsuite/btcutil v1.0.2
github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd
github.com/btcsuite/goleveldb v1.0.0
github.com/btcsuite/snappy-go v1.0.0
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792
github.com/btcsuite/winsvc v1.0.0
github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23
github.com/casbin/casbin/v2 v2.1.2
github.com/cenkalti/backoff v2.2.1+incompatible
github.com/census-instrumentation/opencensus-proto v0.2.1
github.com/cespare/cp v1.1.1
github.com/cespare/xxhash v1.1.0
github.com/cespare/xxhash/v2 v2.1.1
github.com/cheekybits/genny v1.0.0
github.com/clbanning/x2j v0.0.0-20191024224557-825249438eec
github.com/client9/misspell v0.3.4
github.com/cloudflare/cloudflare-go v0.10.2-0.20190916151808-a80f83b9add9
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f
github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd
github.com/coinbase/rosetta-sdk-go v0.6.10
github.com/coreos/bbolt v1.3.2
github.com/coreos/etcd v3.3.10+incompatible
github.com/coreos/go-etcd v2.0.0+incompatible
github.com/coreos/go-semver v0.3.0
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f
github.com/cpuguy83/go-md2man v1.0.10
github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d
github.com/creack/pty v1.1.7
github.com/davecgh/go-spew v1.1.1
github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c
github.com/deckarep/golang-set v1.7.1
github.com/decred/dcrd/lru v1.0.0
github.com/dgraph-io/badger v1.6.1
github.com/dgraph-io/badger/v2 v2.2007.2
github.com/dgraph-io/ristretto v0.0.3
github.com/dgrijalva/jwt-go v3.2.0+incompatible
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954
github.com/docker/docker v1.4.2-0.20180625184442-8e610b2b55bf
github.com/dustin/go-humanize v1.0.0
github.com/eapache/go-resiliency v1.1.0
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21
github.com/eapache/queue v1.1.0
github.com/edsrzf/mmap-go v1.0.0
github.com/elastic/gosigar v0.8.1-0.20180330100440-37f05ff46ffa
github.com/envoyproxy/go-control-plane v0.9.4
github.com/envoyproxy/protoc-gen-validate v0.1.0
github.com/ethereum/go-ethereum v1.9.25 => github.com/ethereum/go-ethereum v1.9.9
github.com/fatih/color v1.10.0
github.com/fjl/memsize v0.0.0-20180929194037-2a09253e352a
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
github.com/flynn/noise v1.0.0
github.com/francoispqt/gojay v1.2.13
github.com/franela/goblin v0.0.0-20200105215937-c9ffbefa60db
github.com/franela/goreq v0.0.0-20171204163338-bcd34c9993f8
github.com/fsnotify/fsnotify v1.4.9
github.com/garslo/gogen v0.0.0-20170307003452-d6ebae628c7c
github.com/gballet/go-libpcsclite v0.0.0-20190607065134-2772fd86a8ff
github.com/ghodss/yaml v1.0.0
github.com/gliderlabs/ssh v0.1.1
github.com/go-check/check v0.0.0-20180628173108-788fd7840127
github.com/go-critic/go-critic v0.4.0
github.com/go-errors/errors v1.0.1
github.com/go-kit/kit v0.10.0
github.com/go-lintpack/lintpack v0.5.2
github.com/go-logfmt/logfmt v0.5.0
github.com/go-ole/go-ole v1.2.1
github.com/go-sql-driver/mysql v1.5.0
github.com/go-stack/stack v1.8.0
github.com/go-toolsmith/astcast v1.0.0
github.com/go-toolsmith/astcopy v1.0.0
github.com/go-toolsmith/astequal v1.0.0
github.com/go-toolsmith/astfmt v1.0.0
github.com/go-toolsmith/astinfo v0.0.0-20180906194353-9809ff7efb21
github.com/go-toolsmith/astp v1.0.0
github.com/go-toolsmith/pkgload v1.0.0
github.com/go-toolsmith/strparse v1.0.0
github.com/go-toolsmith/typep v1.0.0
github.com/gobwas/glob v0.2.3
github.com/gofrs/flock v0.0.0-20190320160742-5135e617513b
github.com/gogo/googleapis v1.1.0
github.com/gogo/protobuf v1.3.2
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7
github.com/golang/mock v1.4.4
github.com/golang/protobuf v1.4.3
github.com/golang/snappy v0.0.1
github.com/golangci/check v0.0.0-20180506172741-cfe4005ccda2
github.com/golangci/dupl v0.0.0-20180902072040-3e9179ac440a
github.com/golangci/errcheck v0.0.0-20181223084120-ef45e06d44b6
github.com/golangci/go-misc v0.0.0-20180628070357-927a3d87b613
github.com/golangci/goconst v0.0.0-20180610141641-041c5f2b40f3
github.com/golangci/gocyclo v0.0.0-20180528134321-2becd97e67ee
github.com/golangci/gofmt v0.0.0-20190930125516-244bba706f1a
github.com/golangci/golangci-lint v1.22.2
github.com/golangci/ineffassign v0.0.0-20190609212857-42439a7714cc
github.com/golangci/lint-1 v0.0.0-20191013205115-297bf364a8e0
github.com/golangci/maligned v0.0.0-20180506175553-b1d89398deca
github.com/golangci/misspell v0.0.0-20180809174111-950f5d19e770
github.com/golangci/prealloc v0.0.0-20180630174525-215b22d4de21
github.com/golangci/revgrep v0.0.0-20180526074752-d9c87f5ffaf0
github.com/golangci/unconvert v0.0.0-20180507085042-28b1c447d1f4
github.com/google/btree v1.0.0
github.com/google/go-cmp v0.5.4
github.com/google/go-github v17.0.0+incompatible
github.com/google/go-querystring v1.0.0
github.com/google/gofuzz v1.0.0
github.com/google/gopacket v1.1.19
github.com/google/martian v2.1.0+incompatible
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57
github.com/google/renameio v0.1.0
github.com/google/uuid v1.2.0
github.com/googleapis/gax-go v2.0.0+incompatible
github.com/googleapis/gax-go/v2 v2.0.3
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1
github.com/gorilla/context v1.1.1
github.com/gorilla/mux v1.8.0
github.com/gorilla/websocket v1.4.2
github.com/gostaticanalysis/analysisutil v0.0.0-20190318220348-4088753ea4d3
github.com/graph-gophers/graphql-go v0.0.0-20191115155744-f33e81362277
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/grpc-ecosystem/grpc-gateway v1.9.5
github.com/gxed/hashland/keccakpg v0.0.1
github.com/gxed/hashland/murmur3 v0.0.1
github.com/harmony-ek/gencodec v0.0.0-20190215044613-e6740dbdd846
github.com/harmony-one/abool v1.0.1
github.com/harmony-one/bls v0.0.6
github.com/harmony-one/taggedrlp v0.1.4
github.com/harmony-one/vdf v0.0.0-20190924175951-620379da8849
github.com/hashicorp/consul/api v1.3.0
github.com/hashicorp/consul/sdk v0.3.0
github.com/hashicorp/errwrap v1.0.0
github.com/hashicorp/go-cleanhttp v0.5.1
github.com/hashicorp/go-immutable-radix v1.0.0
github.com/hashicorp/go-msgpack v0.5.3
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/go-rootcerts v1.0.0
github.com/hashicorp/go-sockaddr v1.0.0
github.com/hashicorp/go-syslog v1.0.0
github.com/hashicorp/go-uuid v1.0.1
github.com/hashicorp/go-version v1.2.0
github.com/hashicorp/go.net v0.0.1
github.com/hashicorp/golang-lru v0.5.4
github.com/hashicorp/hcl v1.0.0
github.com/hashicorp/logutils v1.0.0
github.com/hashicorp/mdns v1.0.0
github.com/hashicorp/memberlist v0.1.3
github.com/hashicorp/serf v0.8.2
github.com/hpcloud/tail v1.0.0
github.com/hudl/fargo v1.3.0
github.com/huin/goupnp v1.0.0
github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150
github.com/inconshreveable/mousetrap v1.0.0
github.com/influxdata/influxdb v1.2.3-0.20180221223340-01288bdb0883
github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d
github.com/ipfs/go-cid v0.0.7
github.com/ipfs/go-datastore v0.4.5
github.com/ipfs/go-detect-race v0.0.1
github.com/ipfs/go-ds-badger v0.2.4
github.com/ipfs/go-ds-leveldb v0.4.2
github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8
github.com/ipfs/go-ipfs-util v0.0.2
github.com/ipfs/go-ipns v0.0.2
github.com/ipfs/go-log v1.0.5
github.com/ipfs/go-log/v2 v2.1.3
github.com/jackpal/gateway v1.0.5
github.com/jackpal/go-nat-pmp v1.0.2
github.com/jbenet/go-cienv v0.1.0
github.com/jbenet/go-temp-err-catcher v0.1.0
github.com/jbenet/goprocess v0.1.4
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
github.com/jessevdk/go-flags v1.4.0
github.com/jmespath/go-jmespath v0.3.0
github.com/jonboulle/clockwork v0.1.0
github.com/jpillora/backoff v1.0.0
github.com/jrick/logrotate v1.0.0
github.com/json-iterator/go v1.1.10
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024
github.com/jtolds/gls v4.20.0+incompatible
github.com/julienschmidt/httprouter v1.3.0
github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d
github.com/karalabe/usb v0.0.0-20190919080040-51dc0efba356
github.com/kisielk/errcheck v1.5.0
github.com/kisielk/gotool v1.0.0
github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23
github.com/klauspost/compress v1.4.1
github.com/klauspost/cpuid v1.2.0
github.com/klauspost/cpuid/v2 v2.0.4
github.com/konsorten/go-windows-terminal-sequences v1.0.3
github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
github.com/kr/pretty v0.2.1
github.com/kr/pty v1.1.8
github.com/kr/text v0.1.0
github.com/kylelemons/godebug v1.1.0
github.com/lib/pq v1.2.0
github.com/libp2p/go-addr-util v0.0.2
github.com/libp2p/go-buffer-pool v0.0.2
github.com/libp2p/go-cidranger v1.1.0
github.com/libp2p/go-conn-security-multistream v0.2.1
github.com/libp2p/go-eventbus v0.2.1
github.com/libp2p/go-flow-metrics v0.0.3
github.com/libp2p/go-libp2p v0.14.4
github.com/libp2p/go-libp2p-asn-util v0.0.0-20200825225859-85005c6cf052
github.com/libp2p/go-libp2p-autonat v0.4.2
github.com/libp2p/go-libp2p-blankhost v0.2.0
github.com/libp2p/go-libp2p-circuit v0.4.0
github.com/libp2p/go-libp2p-connmgr v0.2.4
github.com/libp2p/go-libp2p-core v0.8.6
github.com/libp2p/go-libp2p-crypto v0.1.0
github.com/libp2p/go-libp2p-discovery v0.5.0
github.com/libp2p/go-libp2p-kad-dht v0.12.2
github.com/libp2p/go-libp2p-kbucket v0.4.7
github.com/libp2p/go-libp2p-loggables v0.1.0
github.com/libp2p/go-libp2p-mplex v0.4.1
github.com/libp2p/go-libp2p-nat v0.0.6
github.com/libp2p/go-libp2p-netutil v0.1.0
github.com/libp2p/go-libp2p-noise v0.2.0
github.com/libp2p/go-libp2p-peer v0.2.0
github.com/libp2p/go-libp2p-peerstore v0.2.7
github.com/libp2p/go-libp2p-pnet v0.2.0
github.com/libp2p/go-libp2p-pubsub v0.4.2
github.com/libp2p/go-libp2p-quic-transport v0.10.0
github.com/libp2p/go-libp2p-record v0.1.3
github.com/libp2p/go-libp2p-routing-helpers v0.2.3
github.com/libp2p/go-libp2p-secio v0.2.2
github.com/libp2p/go-libp2p-swarm v0.5.0
github.com/libp2p/go-libp2p-testing v0.4.0
github.com/libp2p/go-libp2p-tls v0.1.3
github.com/libp2p/go-libp2p-transport-upgrader v0.4.2
github.com/libp2p/go-libp2p-xor v0.0.0-20200501025846-71e284145d58
github.com/libp2p/go-libp2p-yamux v0.5.4
github.com/libp2p/go-maddr-filter v0.1.0
github.com/libp2p/go-mplex v0.3.0
github.com/libp2p/go-msgio v0.0.6
github.com/libp2p/go-nat v0.0.5
github.com/libp2p/go-netroute v0.1.6
github.com/libp2p/go-openssl v0.0.7
github.com/libp2p/go-reuseport v0.0.2
github.com/libp2p/go-reuseport-transport v0.0.4
github.com/libp2p/go-sockaddr v0.1.1
github.com/libp2p/go-stream-muxer v0.0.1
github.com/libp2p/go-stream-muxer-multistream v0.3.0
github.com/libp2p/go-tcp-transport v0.2.4
github.com/libp2p/go-ws-transport v0.4.0
github.com/libp2p/go-yamux v1.4.1
github.com/libp2p/go-yamux/v2 v2.2.0
github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-bc2310a04743
github.com/lightstep/lightstep-tracer-go v0.18.1
github.com/logrusorgru/aurora v0.0.0-20181002194514-a7b3b318ed4e
github.com/lucas-clemente/quic-go v0.19.3
github.com/lucasjones/reggen v0.0.0-20180717132126-cdb49ff09d77
github.com/lunixbochs/vtclean v1.0.0
github.com/lyft/protoc-gen-validate v0.0.13
github.com/magiconair/properties v1.8.1
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe
github.com/marten-seemann/qpack v0.2.1
github.com/marten-seemann/qtls v0.10.0
github.com/marten-seemann/qtls-go1-15 v0.1.1
github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd
github.com/matoous/godox v0.0.0-20190911065817-5d6d842e92eb
github.com/mattn/go-colorable v0.1.8
github.com/mattn/go-ieproxy v0.0.0-20190702010315-6dee0af9227d
github.com/mattn/go-isatty v0.0.12
github.com/mattn/go-runewidth v0.0.4
github.com/mattn/goveralls v0.0.2
github.com/matttproud/golang_protobuf_extensions v1.0.1
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b
github.com/microcosm-cc/bluemonday v1.0.1
github.com/miekg/dns v1.1.41
github.com/mikioh/tcp v0.0.0-20190314235350-803a9b46060c
github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b
github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1
github.com/minio/sha256-simd v1.0.0
github.com/mitchellh/cli v1.0.0
github.com/mitchellh/go-homedir v1.1.0
github.com/mitchellh/go-ps v0.0.0-20190716172923-621e5597135b
github.com/mitchellh/go-testing-interface v1.0.0
github.com/mitchellh/gox v0.4.0
github.com/mitchellh/iochan v1.0.0
github.com/mitchellh/mapstructure v1.3.3
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
github.com/modern-go/reflect2 v1.0.1
github.com/mozilla/tls-observatory v0.0.0-20190404164649-a3c1b6cfecfd
github.com/mr-tron/base58 v1.2.0
github.com/multiformats/go-base32 v0.0.3
github.com/multiformats/go-base36 v0.1.0
github.com/multiformats/go-multiaddr v0.3.3
github.com/multiformats/go-multiaddr-dns v0.3.1
github.com/multiformats/go-multiaddr-fmt v0.1.0
github.com/multiformats/go-multiaddr-net v0.2.0
github.com/multiformats/go-multibase v0.0.3
github.com/multiformats/go-multihash v0.0.15
github.com/multiformats/go-multistream v0.2.2
github.com/multiformats/go-varint v0.0.6
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
github.com/naoina/go-stringutil v0.1.0
github.com/naoina/toml v0.1.2-0.20170918210437-9fafd6967416
github.com/natefinch/lumberjack v2.0.0+incompatible
github.com/nats-io/jwt v0.3.2
github.com/nats-io/nats-server/v2 v2.1.2
github.com/nats-io/nats.go v1.9.1
github.com/nats-io/nkeys v0.1.3
github.com/nats-io/nuid v1.0.1
github.com/nbutton23/zxcvbn-go v0.0.0-20180912185939-ae427f1e4c1d
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab
github.com/neilotoole/errgroup v0.1.5
github.com/nxadm/tail v1.4.4
github.com/oklog/oklog v0.3.2
github.com/oklog/run v1.0.0
github.com/oklog/ulid v1.3.1
github.com/olekukonko/tablewriter v0.0.2-0.20190409134802-7e037d187b0c
github.com/onsi/ginkgo v1.14.0
github.com/onsi/gomega v1.10.1
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
github.com/opentracing-contrib/go-observer v0.0.0-20170622124052-a52f23424492
github.com/opentracing/basictracer-go v1.0.0
github.com/opentracing/opentracing-go v1.2.0
github.com/openzipkin-contrib/zipkin-go-opentracing v0.4.5
github.com/openzipkin/zipkin-go v0.2.2
github.com/pact-foundation/pact-go v1.0.4
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c
github.com/pborman/uuid v1.2.0
github.com/pelletier/go-toml v1.9.3
github.com/performancecopilot/speed v3.0.0+incompatible
github.com/peterh/liner v1.1.1-0.20190123174540-a2c9a5303de7
github.com/pierrec/lz4 v2.0.5+incompatible
github.com/pkg/errors v0.9.1
github.com/pkg/profile v1.2.1
github.com/pmezard/go-difflib v1.0.0
github.com/posener/complete v1.1.1
github.com/prometheus/client_golang v1.10.0
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.18.0
github.com/prometheus/procfs v0.6.0
github.com/prometheus/tsdb v0.7.1
github.com/quasilyte/go-consistent v0.0.0-20190521200055-c6f3937de18c
github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0
github.com/rjeczalik/notify v0.9.2
github.com/robertkrimen/otto v0.0.0-20170205013659-6a77b7cbc37d
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af
github.com/rogpeppe/go-internal v1.3.0
github.com/rs/cors v1.7.0
github.com/rs/xhandler v0.0.0-20160618193221-ed27b6fd6521
github.com/rs/xid v1.2.1
github.com/rs/zerolog v1.18.0
github.com/russross/blackfriday v1.5.2
github.com/russross/blackfriday/v2 v2.0.1
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f
github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529
github.com/securego/gosec v0.0.0-20191002120514-e680875ea14d
github.com/segmentio/fasthash v1.0.3
github.com/sergi/go-diff v1.0.0
github.com/shirou/gopsutil v0.0.0-20190901111213-e4ec7b275ada
github.com/shirou/w32 v0.0.0-20160930032740-bb4de0191aa4
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82
github.com/shurcooL/sanitized_anchor_name v1.0.0
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133
github.com/sirupsen/logrus v1.6.0
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d
github.com/smartystreets/goconvey v1.6.4
github.com/smola/gocompat v0.2.0
github.com/soheilhy/cmux v0.1.4
github.com/sony/gobreaker v0.4.1
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
github.com/sourcegraph/go-diff v0.5.1
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572
github.com/spaolacci/murmur3 v1.1.0
github.com/spf13/afero v1.1.2
github.com/spf13/cast v1.3.0
github.com/spf13/cobra v0.0.5
github.com/spf13/jwalterweatherman v1.0.0
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.6.1
github.com/src-d/envconfig v1.0.0
github.com/status-im/keycard-go v0.0.0-20190316090335-8537d3370df4
github.com/steakknife/bloomfilter v0.0.0-20180922174646-6819c0d2a570
github.com/steakknife/hamming v0.0.0-20180906055917-c99c65617cd3
github.com/streadway/amqp v0.0.0-20190827072141-edfb9018d271
github.com/streadway/handy v0.0.0-20190108123426-d5acb3125c2a
github.com/stretchr/objx v0.2.0
github.com/stretchr/testify v1.7.0
github.com/subosito/gotenv v1.2.0
github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07
github.com/tidwall/gjson v1.6.7
github.com/tidwall/match v1.0.3
github.com/tidwall/pretty v1.0.2
github.com/tidwall/sjson v1.1.4
github.com/timakin/bodyclose v0.0.0-20190930140734-f7f2e9bca95e
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5
github.com/tommy-muehle/go-mnd v1.1.1
github.com/tyler-smith/go-bip39 v1.0.2
github.com/ugorji/go v1.1.4
github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8
github.com/ultraware/funlen v0.0.2
github.com/ultraware/whitespace v0.0.4
github.com/urfave/cli v1.22.1
github.com/uudashr/gocognit v1.0.1
github.com/valyala/bytebufferpool v1.0.0
github.com/valyala/fasthttp v1.2.0
github.com/valyala/quicktemplate v1.2.0
github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a
github.com/viant/assertly v0.4.8
github.com/viant/toolbox v0.24.0
github.com/vmihailenco/msgpack/v5 v5.1.4
github.com/vmihailenco/tagparser v0.1.2
github.com/wangjia184/sortedset v0.0.0-20160527075905-f5d03557ba30
github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1
github.com/whyrusleeping/go-logging v0.0.1
github.com/whyrusleeping/mafmt v1.2.8
github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9
github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7
github.com/whyrusleeping/timecache v0.0.0-20160911033111-cfcb2f1abfee
github.com/wsddn/go-ecdh v0.0.0-20161211032359-48726bab9208
github.com/x-cray/logrus-prefixed-formatter v0.5.2
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77
github.com/ybbus/jsonrpc v2.1.2+incompatible
github.com/yuin/goldmark v1.2.1
github.com/zenazn/goji v0.9.0
go.etcd.io/bbolt v1.3.3
go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738
go.opencensus.io v0.23.0
go.uber.org/atomic v1.7.0
go.uber.org/goleak v1.0.0
go.uber.org/multierr v1.6.0
go.uber.org/ratelimit v0.1.0
go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee
go.uber.org/zap v1.16.0
go4.org v0.0.0-20180809161055-417644f6feb5
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2
golang.org/x/exp v0.0.0-20190121172915-509febef88a4
golang.org/x/lint v0.0.0-20200302205851-738671d3881b
golang.org/x/mod v0.3.0
golang.org/x/net v0.0.0-20210423184538-5f58ad60dda6
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
golang.org/x/sys v0.0.0-20210426080607-c94f62235c83
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1
golang.org/x/text v0.3.6
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba
golang.org/x/tools v0.0.0-20210106214847-113979e3529a
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
google.golang.org/api v0.3.1
google.golang.org/appengine v1.4.0
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013
google.golang.org/grpc v1.33.2
google.golang.org/protobuf v1.25.0
gopkg.in/alecthomas/kingpin.v2 v2.2.6
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
gopkg.in/cheggaaa/pb.v1 v1.0.25
gopkg.in/errgo.v2 v2.1.0
gopkg.in/fsnotify.v1 v1.4.7
gopkg.in/gcfg.v1 v1.2.3
gopkg.in/inf.v0 v0.9.1
gopkg.in/ini.v1 v1.51.0
gopkg.in/natefinch/lumberjack.v2 v2.0.0
gopkg.in/natefinch/npipe.v2 v2.0.0-20160621034901-c1b8fa8bdcce
gopkg.in/olebedev/go-duktape.v3 v3.0.0-20200619000410-60c24ae608a6
gopkg.in/resty.v1 v1.12.0
gopkg.in/sourcemap.v1 v1.0.5
gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d
gopkg.in/src-d/go-log.v1 v1.0.1
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
gopkg.in/urfave/cli.v1 v1.20.0
gopkg.in/warnings.v0 v0.1.2
gopkg.in/yaml.v2 v2.3.0
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
gotest.tools v2.2.0+incompatible
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
honnef.co/go/tools v0.0.1-2020.1.5
mvdan.cc/interfacer v0.0.0-20180901003855-c20040233aed
mvdan.cc/lint v0.0.0-20170908181259-adc824a0674b
mvdan.cc/unparam v0.0.0-20190720180237-d51796306d8f
rsc.io/quote/v3 v3.1.0
rsc.io/sampler v1.3.0
sigs.k8s.io/yaml v1.1.0
sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0
sourcegraph.com/sourcegraph/go-diff v0.5.0
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4
Stebalien commented 3 years ago

Ok, so you definitely don't have the memory leak I was thinking about (was fixed in go-tcp-transport v0.2.4).

Do you have a goroutine pprof profile (ideally with debug=2) by any chance?

Also, which DHT are you using? Are you using the "fullrt" DHT, or the regular DHT?

Finally, how many connections do you have open (can just use netstat or ss for a rough estimate)? Libp2p doesn't set connection limits by default. I ask because the DHT will (currently) keep up-to one stream per peer open in the query logic (which is something we should probably not be doing).

Discovery

I assume you mean go-libp2p-discovery. Are you using this in conjunction with pubsub? I don't think this will be relevant, but I want to make sure we're not missing something.

AlexiaChen commented 3 years ago

For you last question I can answer now:

Yes, go-libp2p-discovery is using in conjunction with libp2p-pubsub

How to check which DHT ? here is our code of creating DHT Discovery:

// NewDHTDiscovery creates a new dhtDiscovery that implements Discovery interface.
func NewDHTDiscovery(host libp2p_host.Host, opt DHTConfig) (Discovery, error) {
    opts, err := opt.getLibp2pRawOptions()
    if err != nil {
        return nil, err
    }
    ctx, cancel := context.WithCancel(context.Background())
    dht, err := libp2p_dht.New(ctx, host, opts...)
    if err != nil {
        return nil, err
    }
    d := libp2p_dis.NewRoutingDiscovery(dht)

    logger := utils.Logger().With().Str("module", "discovery").Logger()
    return &dhtDiscovery{
        dht:    dht,
        disc:   d,
        host:   host,
        opt:    opt,
        logger: logger,
        ctx:    ctx,
        cancel: cancel,
    }, nil
}

Do you have a goroutine pprof profile (ideally with debug=2) by any chance?

give me movre time to check and I will inform you

Wondertan commented 3 years ago

@AlexiaChen, you mentioned using go-kad-dht 0.12.2 which was introduced much earlier than the TCP leak issue was fixed. Are you sure that you didn't update your dependencies before providing go list list -m all results? Also, we discussed this issue with the Harmony team before the TCP fix, so I still think that can be related to TCP. Can you please verify you didn't update your deps after reporting the issue? Also, try reproducing the leak with the latest deps.

AlexiaChen commented 3 years ago

@Wondertan

kad-dht https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.12.2 is published on 19 Jun. it not very earlier. The problem is actually happening on the harmony mainnet (production environment). This is definitely using the latest deps, but there is still this issue, which only started happening when libp2p's pubsub used Discovery. commit is here https://github.com/harmony-one/harmony/commit/3d6858aba53897aa42b219ef75f8a67914646e96# it added new file discovery.go

Wondertan commented 3 years ago

@AlexiaChen, while https://github.com/libp2p/go-tcp-transport/releases/tag/v0.2.4 was published on Jul 15 and that's almost a month, so here is a concern.

AlexiaChen commented 3 years ago

@Wondertan But we do not directy use this go-tcp-transport. only packages related to libp2p used here in go.mod: https://github.com/harmony-one/harmony/blob/b34d06b2690395ca0ab64c34be72484ed92249d0/go.mod#L33-L38

AlexiaChen commented 3 years ago

indirectly use gp-tcp-transport on github.com/libp2p/go-tcp-transport v0.2.4 show from go list -m all(harmony main branch), you can recheck the log I mentioned above. So I need to check the version of tcp-transport on mainnet at first . Thanks

AlexiaChen commented 3 years ago

@Stebalien @Wondertan Sorry, I made a mistake, I rechecked the version of go-tcp-transport on mainnet(production env) is v0.2.1, the version has fixed memory leak ?

the log from go list -m all I mentioned above execute on latest main branch of harmony, its not consistent with production env. Sorry.

mathxh@MathxH:~/go/src/github.com/harmony-one/harmony$ git reset --hard  004a5867e8b067ae29217dc2efd51f82711fb5e1
HEAD is now at 004a5867 launch vrf and 3 governance proposal at epoch 631
mathxh@MathxH:~/go/src/github.com/harmony-one/harmony$ make clean
rm -rf ./tmp_log*
rm -rf ./.dht*
rm -rf ./db-*
rm -rf ./latest
rm -f ./*.rlp
rm -rf ~/rpmbuild
rm -f coverage.txt
mathxh@MathxH:~/go/src/github.com/harmony-one/harmony$ go list -m all | grep tcp-transport
go: github.com/libp2p/go-libp2p@v0.14.0: missing go.sum entry; to add it:
        go mod download github.com/libp2p/go-libp2p
mathxh@MathxH:~/go/src/github.com/harmony-one/harmony$ go mod tidy
mathxh@MathxH:~/go/src/github.com/harmony-one/harmony$ go list -m all | grep tcp-transport
github.com/libp2p/go-tcp-transport v0.2.1
Wondertan commented 3 years ago

@AlexiaChen, v0.2.1 does not have a fix for a memory leak, but it seems like the issue itself was introduced in v0.2.2, so the version you were running is not affected by the TCP leak. However, it is better to wait for Steven to verify such reasoning, as here knows better.

Stebalien commented 3 years ago

Yeah, v0.2.1 should be safe as well.

godeamon commented 2 years ago

@Stebalien @Wondertan @AlexiaChen Hello, I also encountered the same problem, how did you solve it? Here is my lib version from go list -m all cmd: github.com/libp2p/go-tcp-transport v0.2.1

Stebalien commented 2 years ago

@godeamon please update. You're over 2 years out of date.

Stebalien commented 2 years ago

We've recently changed how notifications get delivered, which likely fixed this issue. I'm going to close this unless someone finds a memory leak in the current go-libp2p version.