Memory leaks and network throttling after 4 days

botaniver commented 6 years ago

Current openbazaar-go takes a huge amount of memory (almost 2 GB) once left running for just 3 or 4 days. Furthermore, once that happens, all other network connections get throttled down to 40-50 kB/s.

I'm not sure whether the network throttling is due simply to lack of memory (openbazaar-go eating all RAM and swap) or some other resource exhaustion (actual bandwidth being used by openbazaar-go). I am investigating that and I'll report here. The strange part though, is that this happened consistently on two different machines, running Debian 9 and Ubuntu 16.04 respectively. On both I experienced exactly the same throttling to 40-50 kB/s. openbazaar-go is running with a non-root user, and even root processes get throttled.

I've been experiencing this since version v0.10, throughout v0.10.1 and now with v0.11.

christroutner commented 6 years ago

This hasn't been my experience. I'm running two separate stores on separate Digital Ocean Droplet VPS servers running Ubuntu 16.04. I haven't had any problem with memory usage or throttling.

cpacia commented 6 years ago

We haven't really seen that either. We've got instances running for months without issue.

botaniver commented 6 years ago

I restarted my node when I opened this issue. It is now taking 400 MB of memory. I'll come back with more data once it reaches 1 GB, which usually takes just a few days.

botaniver commented 6 years ago

Got it to reproduce the issue. Current node stats:

taking 780 MB of resident memory,
spun 51 threads,
averaging 10% CPU use (single core VPS),
constantly sending 100 KB/s and receiving 40 KB/s (nethogs),
all other network connections are throttled to 80 KB/s, although the VPS is supposed to be 100/100 MBit/s (stopping openbazar-go will fix this).

Running openbazaar-go v0.11.0 on a fully up-to-date Ubuntu 16.04.3 LTS.

Logs show nothing relevant:

2018/02/13 09:01:15 http: TLS handshake error from [::1]:4001: EOF 2018/02/13 09:01:15 http: TLS handshake error from 127.0.0.1:4001: EOF

Is there any extra info you would like me to gather before restarting the node?

Current config:

{
   "API": {
      "HTTPHeaders": null
   },
   "Addresses": {
      "API": "",
      "Gateway": "/ip4/0.0.0.0/tcp/4002",
      "Swarm": [
         "/ip4/0.0.0.0/tcp/4001",
         "/ip6/::/tcp/4001",
         "/ip4/0.0.0.0/tcp/9005/ws",
         "/ip6/::/tcp/9005/ws"
      ]
   },
   "Bootstrap": [
      "/ip4/107.170.133.32/tcp/4001/ipfs/QmUZRGLhcKXF1JyuaHgKm23LvqcoMYwtb9jmh8CkP4og3K",
      "/ip4/139.59.174.197/tcp/4001/ipfs/QmcCoBtYyduyurcLHRF14QhhA88YojJJpGFuMHoMZuU8sc",
      "/ip4/139.59.6.222/tcp/4001/ipfs/QmRDcEDK9gSViAevCHiE6ghkaBCU7rTuQj4BDpmCzRvRYg",
      "/ip4/46.101.198.170/tcp/4001/ipfs/QmePWxsFT9wY3QuukgVDB7XZpqdKhrqJTHTXU7ECLDWJqX"
   ],
   "Bootstrap-testnet": [
      "/ip4/165.227.117.91/tcp/4001/ipfs/Qmaa6De5QYNqShzPb9SGSo8vLmoUte8mnWgzn4GYwzuUYA"
   ],
   "DataSharing": {
      "AcceptStoreRequests": false,
      "PushTo": [
         "QmY8puEnVx66uEet64gAf4VZRo7oUyMCwG6KdB9KM92EGQ",
         "QmPPg2qeF3n2KvTRXRZLaTwHCw8JxzF4uZK93RfMoDvf2o",
         "QmPPegaeM4rXfQDF3uu784d93pLEzV8A4zXU7akEgYnTFd"
      ]
   },
   "Datastore": {
      "BloomFilterSize": 0,
      "GCPeriod": "1h",
      "HashOnRead": false,
      "Spec": {
         "mounts": [
            {
               "child": {
                  "path": "blocks",
                  "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
                  "sync": true,
                  "type": "flatfs"
               },
               "mountpoint": "/blocks",
               "prefix": "flatfs.datastore",
               "type": "measure"
            },
            {
               "child": {
                  "compression": "none",
                  "path": "datastore",
                  "type": "levelds"
               },
               "mountpoint": "/",
               "prefix": "leveldb.datastore",
               "type": "measure"
            }
         ],
         "type": "mount"
      },
      "StorageGCWatermark": 90,
      "StorageMax": "10GB"
   },
   "Discovery": {
      "MDNS": {
         "Enabled": false,
         "Interval": 10
      }
   },
   "Dropbox-api-token": "",
   "Experimental": {
      "FilestoreEnabled": false,
      "Libp2pStreamMounting": false,
      "ShardingEnabled": false
   },
   "Gateway": {
      "HTTPHeaders": null,
      "PathPrefixes": [],
      "RootRedirect": "",
      "Writable": false
   },
   "Identity": {
      "PeerID": ""
   },
   "Ipns": {
      "BackUpAPI": "https://gateway.ob1.io/ob/ipns/",
      "QuerySize": 1,
      "RecordLifetime": "7d",
      "RepublishPeriod": "24h",
      "ResolveCacheSize": 128,
      "UsePersistentCache": true
   },
   "JSON-API": {
      "AllowedIPs": [],
      "Authenticated": true,
      "CORS": null,
      "Enabled": true,
      "HTTPHeaders": null,
      "Password": "-redacted-",
      "SSL": true,
      "SSLCert": "-redacted-",
      "SSLKey": "-redacted-",
      "Username": "-redacted-"
   },
   "Mounts": {
      "FuseAllowOther": false,
      "IPFS": "/ipfs",
      "IPNS": "/ipns"
   },
   "Reprovider": {
      "Interval": ""
   },
   "RepublishInterval": "24h",
   "Resolvers": {
      ".eth": "",
      ".id": "https://resolver.onename.com/"
   },
   "SupernodeRouting": {
      "Servers": null
   },
   "Swarm": {
      "AddrFilters": null,
      "DisableBandwidthMetrics": false,
      "DisableNatPortMap": false
   },
   "Tor-config": {
      "Password": "",
      "TorControl": ""
   },
   "Tour": {
      "Last": ""
   },
   "Wallet": {
      "Binary": "",
      "FeeAPI": "https://btc.fees.openbazaar.org",
      "HighFeeDefault": 160,
      "LowFeeDefault": 20,
      "MaxFee": 2000,
      "MediumFeeDefault": 60,
      "RPCPassword": "",
      "RPCUser": "",
      "TrustedPeer": "",
      "Type": "spvwallet"
   }
}

% ps aux | grep openbazaar
root     16534 16.5 76.5 1918520 774912 ?      Ssl  Feb09 1244:20 /usr/local/bin/openbazaar-go start

% cat /proc/16534/status
Name:   openbazaar-go
State:  S (sleeping)
Tgid:   16534
Ngid:   0
Pid:    16534
PPid:   1
TracerPid:  0
Uid:    0   0   0   0
Gid:    0   0   0   0
FDSize: 8192
Groups: 
NStgid: 16534
NSpid:  16534
NSpgid: 16534
NSsid:  16534
VmPeak:  1918520 kB
VmSize:  1918520 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:    820580 kB
VmRSS:    774772 kB
VmData:  1876508 kB
VmStk:       132 kB
VmExe:     26372 kB
VmLib:      2040 kB
VmPTE:      2112 kB
VmPMD:        28 kB
VmSwap:    88860 kB
HugetlbPages:          0 kB
Threads:    56
SigQ:   0/3585
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: fffffffe7bfa7a25
SigIgn: 0000000000000000
SigCgt: ffffffffffc1feff
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
Seccomp:    0
Cpus_allowed:   1
Cpus_allowed_list:  0
Mems_allowed:   00000000,00000001
Mems_allowed_list:  0
voluntary_ctxt_switches:    129
nonvoluntary_ctxt_switches: 707

cpacia commented 6 years ago

I'm going to have to try to replicate this. We're also planning on rebasing IPFS pretty soon so it's possible any issues could be fixed with the rebase. I will probably wait to test this until after the rebase.

botaniver commented 6 years ago

I can test it myself once there is a release with the rebased IPFS code. I'll let you know it the problem persists.

OpenBazaar / openbazaar-go

Memory leaks and network throttling after 4 days #858