Kernel Memory Leak: possible conflict between LuaJIT or OpenResty and the consul server agent

jmealo commented 8 years ago

This is not related to #1060.

I'm running the official linux x64 binary release of v0.5.2 on Ubuntu 14.04. I have a single datacenter cluster with seven nodes (all running as server agents).

Here is an example of one of the misbehaving server agents: cat /proc/$(pidof consul)/status

root@redacted:/proc# cat /proc/$(pidof consul)/status
Name:   consul
State:  S (sleeping)
Tgid:   1003
Ngid:   0
Pid:    1003
PPid:   1
TracerPid:  0
Uid:    106 106 106 106
Gid:    114 114 114 114
FDSize: 64
Groups: 114 
VmPeak: 68384600 kB
VmSize: 35089012 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:    487456 kB
VmRSS:    485788 kB
VmData:   460908 kB
VmStk:       136 kB
VmExe:     12096 kB
VmLib:      2168 kB
VmPTE:      1284 kB
VmSwap:        0 kB
Threads:    9
SigQ:   0/7782
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: ffffffffffc1feff
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000001fffffffff
Seccomp:    0
Cpus_allowed:   1
Cpus_allowed_list:  0
Mems_allowed:   00000000,00000001
Mems_allowed_list:  0
voluntary_ctxt_switches:    130378
nonvoluntary_ctxt_switches: 372223

consul --version:

root@redacted:/proc# consul --version
Consul v0.5.2
Consul Protocol: 2 (Understands back to: 1)

/etc/consul.d/server/config.json

{
    "server": true,
    "datacenter": "redacted",
    "data_dir": "/var/consul",
    "encrypt": "redacted",
    "log_level": "INFO",
    "enable_syslog": true,
    "retry_join": [
        "redacted",
        "redacted",
        "redacted",
        "redacted",
        "redacted",
        "redacted",
    ],
    "bind_addr": "redacted",
    "ui_dir": "/var/www/consul"
}

/etc/consul.d/server/nginx.json

{
        "service": {
                "name": "nginx",
                "tags": ["www"],
                "port": 80
        }
}

uname -a:

Linux redacted 3.13.0-68-generic #111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

slackpad commented 8 years ago

Hi @jmealo the best way to see where the memory is going is to set the enable_debug option on one of your servers and then look at http://localhost:8500/debug/pprof/heap?debug=1 on that server to see the heap profile. Can you post the top few entries from there into a gist?

Consul 0.6 has a number of fixes to improve memory usage, including a new in-memory database that's more efficient as well as automatic reduction of large receive buffers for idle connections to the servers (these can definitely hit a few hundred megabytes if your servers have gotten a huge burst of RPC traffic).

jmealo commented 8 years ago

@slackpad I'll definitely be documenting this as the OOM killer just took out one of my load balancers. This configuration worked flawlessly on 512mb ram prior to introducing Consul. Now even with a 1024mb box consul is able to take the whole node down.

jmealo commented 8 years ago

After running only 6 minutes, here's the output of /debug/pprof/heap?debug=1:

node-2.txt https://gist.github.com/jmealo/444e82a48504441f85c5

slackpad commented 8 years ago

That one looks pretty normal with objects built up in the state store after initialization and some other startup items - 10 megs worth of stuff.

jmealo commented 8 years ago

@slackpad: I upgraded to 0.60 and the problem persists. Here's the output of /debug/pprof/heap?debug=1 after running for 8 hours 38 minutes.

output.txt https://gist.github.com/jmealo/fbae77eca68cca535fb9

Output of free -tm:

             total       used       free     shared    buffers     cached
Mem:           994        901         92          0          2         41
-/+ buffers/cache:        857        136
Swap:         2047         10       2037
Total:        3042        912       2129

Output of cat /proc/meminfo:

MemTotal:        1017860 kB
MemFree:           94176 kB
Buffers:            2364 kB
Cached:            42920 kB
SwapCached:         2308 kB
Active:            25224 kB
Inactive:          30900 kB
Active(anon):       5296 kB
Inactive(anon):     5604 kB
Active(file):      19928 kB
Inactive(file):    25296 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2097148 kB
SwapFree:        2086240 kB
Dirty:                32 kB
Writeback:             0 kB
AnonPages:          9084 kB
Mapped:             6140 kB
Shmem:                28 kB
Slab:              26384 kB
SReclaimable:       9884 kB
SUnreclaim:        16500 kB
KernelStack:         656 kB
PageTables:         2540 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2606076 kB
Committed_AS:      81976 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        9440 kB
VmallocChunk:   34359721212 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       47096 kB
DirectMap2M:     1001472 kB
DirectMap1G:           0 kB

Output of slabtop -o:

 Active / Total Objects (% used)    : 109067 / 110719 (98.5%)
 Active / Total Slabs (% used)      : 4993 / 4993 (100.0%)
 Active / Total Caches (% used)     : 62 / 99 (62.6%)
 Active / Total Size (% used)       : 25351.91K / 26071.38K (97.2%)
 Minimum / Average / Maximum Object : 0.01K / 0.24K / 15.75K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
 33936  33687  99%    0.25K   2121       16      8484K kmalloc-256            
 14904  14904 100%    0.11K    414       36      1656K sysfs_dir_cache        
 13881  13240  95%    0.19K    661       21      2644K dentry                 
  8640   8275  95%    0.06K    135       64       540K kmalloc-64             
  7364   7364 100%    0.57K    526       14      4208K inode_cache            
  3591   3448  96%    0.19K    171       21       684K kmalloc-192            
  3584   3584 100%    0.01K      7      512        28K kmalloc-8              
  2550   2550 100%    0.05K     30       85       120K shared_policy_node     
  2304   2304 100%    0.02K      9      256        36K kmalloc-16             
  2240   2240 100%    0.07K     40       56       160K Acpi-ParseExt          
  2040   2040 100%    0.04K     20      102        80K ext4_extent_status     
  1920   1920 100%    0.03K     15      128        60K kmalloc-32             
  1677   1677 100%    0.10K     43       39       172K buffer_head            
  1576   1515  96%    0.96K    197        8      1576K ext4_inode_cache       
  1456   1456 100%    0.07K     26       56       104K anon_vma               
  1024   1024 100%    0.12K     32       32       128K kmalloc-128            
  1008   1002  99%    0.63K     84       12       672K proc_inode_cache       
   744    744 100%    0.66K     62       12       496K shmem_inode_cache      
   602    602 100%    0.55K     43       14       344K radix_tree_node        
   588    588 100%    0.09K     14       42        56K kmalloc-96             
   560    510  91%    0.50K     70        8       280K kmalloc-512            
   544    498  91%    1.00K     68        8       544K kmalloc-1024           
   408    408 100%    0.04K      4      102        16K Acpi-Namespace         
   288    288 100%    0.11K      8       36        32K jbd2_journal_head      
   270    270 100%    0.13K      9       30        36K ext4_allocation_context
   256    218  85%    2.00K     32        8       512K kmalloc-2048           
   256    256 100%    0.02K      1      256         4K jbd2_revoke_table_s    
   170    170 100%    0.02K      1      170         4K numa_policy            
   153    153 100%    0.08K      3       51        12K Acpi-State             
   152    152 100%    0.81K      8       19       128K task_xstate            
   135    135 100%    1.06K      9       15       144K signal_cache           
   128    128 100%    0.06K      2       64         8K kmem_cache_node        
   128    128 100%    0.03K      1      128         4K jbd2_revoke_record_s   
   125     92  73%    6.00K     25        5       800K task_struct            
   120    120 100%    2.06K      8       15       256K idr_layer_cache        
   112    112 100%    0.25K      7       16        28K kmem_cache             
   108    108 100%    0.62K      9       12        72K sock_inode_cache       
   105    105 100%    2.06K      7       15       224K sighand_cache          
    90     90 100%    0.88K     10        9        80K mm_struct              
    88     68  77%    4.00K     11        8       352K kmalloc-4096           
    85     85 100%    0.05K      1       85         4K jbd2_journal_handle    
    78     78 100%    0.10K      2       39         8K blkdev_ioc             
    73     73 100%    0.05K      1       73         4K ip_fib_trie            
    70     70 100%    0.38K      7       10        28K blkdev_requests        
    68     68 100%    0.12K      2       34         8K fsnotify_event         
    64     64 100%    0.06K      1       64         4K ext4_free_data         
    60     60 100%    0.62K      5       12        40K files_cache            
    56     56 100%    0.07K      1       56         4K ext4_io_end            
    48     48 100%    0.25K      3       16        12K tw_sock_TCP            
    42     42 100%    2.21K      3       14        96K blkdev_queue           
    38     38 100%    0.81K      2       19        32K bdev_cache             
    36     36 100%    0.31K      3       12        12K mnt_cache              
    36     36 100%    1.75K      4        9        64K TCP                    
    25     25 100%    0.16K      1       25         4K sigqueue               
    24     24 100%    8.00K      6        4       192K kmalloc-8192           
    18     18 100%    0.88K      2        9        16K UDP                    
    16     16 100%    0.25K      1       16         4K dquot                  
    15     15 100%    1.06K      1       15        16K UDPv6                  
    13     13 100%    0.59K      1       13         8K hugetlbfs_inode_cache

Output of vmstat:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0  10904  83964   3044  52540    2    3   220   458  235  537  1  1 98  0  0

slackpad commented 8 years ago

Hi @jmealo - that latest dump looks pretty normal as well, and looking at the end # Sys = 57092344 that looks like the Go runtime thinks it's using on the order of 60 megabytes, which is super reasonable for a Consul server. Can you run ps -eo vsz,rss,comm | grep consul so we can see what the RSS value is for Consul?

jmealo commented 8 years ago

This is on a different node, still it's a 1GB node running NGINX and Consul:

consul:

ps -eo vsz,rss,comm | grep consul
496180 16720 consul-alerts
 26316 11324 consul
 17860  5784 consul
1089032 75412 consul

nginx:

ps -eo vsz,rss,comm | grep nginx
 71796  3720 nginx
 72904  5308 nginx
 71796  4008 nginx

I finally have some cached memory on this one, rather than it all disappearing. I had my provider switch one of the nodes to a different hypervisor. If the problem doesn't persist I'll close the ticket. So far it's been like clockwork.

slackpad commented 8 years ago

Ok cool - yeah these numbers all look pretty good - thanks for providing all the debug info!

jmealo commented 8 years ago

@slackpad We just hit 92% memory usage again, attached are the output of all of the commands that I've provided before as well as the ps output that you requested for consul, nginx, and all processes.

consul_debug.txt free.txt meminfo.txt nginx.ps.txt ps.all.txt slabtop.txt status.txt vmstat.txt consul.ps.txt

slackpad commented 8 years ago

Hmm - I still don't see anything that points to Consul (or any of your other processes for that matter):

1113600 37648 consul

That's showing an RSS for Consul of about 38 MB. And that roughly matches your debug logs:

# Sys = 52492536

Though from looking at the debug log it had just done a Raft snapshot, so it may have freed up some stuff. It might be interesting to run your ps.all in a loop every few seconds and save that off into a log so we can see if it's maybe building up a bunch of memory and then releasing it when it compacts the Raft log, though looking at the recent history in the debug log there's not a ton of stuff in there.

What your access pattern like against Consul - do you have periods of huge write volumes in a short time window?

jmealo commented 8 years ago

@slackpad: I don't think so, we're not heavily using it right now. We have 7 nodes and 48 health checks total.

jmealo commented 8 years ago

Judging by the fact that we only have this issue on the load balancers, I'm getting ready to close this ticket. All 7 nodes are running as server agents with one of the load balancers typically being the leader. Would being a leader cause a drastic change in shared memory usage?

Does this look like a kernel issue? I just don't know how to figure out where the memory is going.

slackpad commented 8 years ago

Not drastic - all the servers will make the same changes to their data store. Writes and some queries will go to the leader so it'll have more active connections and a little more GC churn, but shouldn't be way out of family with the other Consul servers.

jmealo commented 8 years ago

On the bright side, the memory usage seems to have improved greatly in 0.60x. :+1: If anyone has any ideas of where to go from here I'd greatly appreciate it.

jmealo commented 8 years ago

@slackpad Thanks for all of your help/patience. This is what the memory usage looks like, it presents itself like a classic memory leak, however, it doesn't seem to be based on requests or traffic, but rather time.

screen shot 2015-12-10 at 1 46 32 pm

Due to consul running health checks at regular intervals, that might create enough traffic to influence the graph.

slackpad commented 8 years ago

If the health checks have text output that varies from run to run (like timestamps) and/or a lot of output, that could cause a lot of churn in the Raft log; even if the checks are always passing. That might be something to check.

If there was a lot of churn in the health checks (and if they run pretty often) then it could build up a big Raft log that eventually gets compacted during a snapshot. The Raft log would hold each delta change with the new check output, and when that gets compacted you'd only be left with the latest update to the check in the in-memory data store. I'm not sure that the big Raft log would occupy a ton of RAM though, so this is kind of speculative.

One final thing is you could dump telemetry on one of the Consul servers and we could look for fishiness there - https://www.consul.io/docs/agent/telemetry.html. Be aware that this might include some service names and other info, so be careful if you post this.

jmealo commented 8 years ago

@slackpad: It certainly seems like it is raft using up the memory (see output below). I'm not sure why pmap is the only tool that knows that it is consul consuming the memory. I will dump telemetry if I'm able to sanitize it easily.

root@redacted:~# pmap -X $(pidof consul)
991:   consul agent -config-dir /etc/consul.d/server
         Address Perm   Offset Device  Inode    Size   Rss   Pss Referenced Anonymous Swap Locked Mapping
        00400000 r-xp 00000000  fd:01 922462    6480  1676  1676       1676         0    0      0 consul
        00a54000 r--p 00654000  fd:01 922462    6820  1616  1616       1616         0    0      0 consul
        010fd000 rw-p 00cfd000  fd:01 922462     132    72    72         72        36    0      0 consul
        0111e000 rw-p 00000000  00:00      0     184   116   116        116       116    0      0 
      c000000000 rw-p 00000000  00:00      0      32    32    32         32        32    0      0 
      c81ff04000 rw-p 00000000  00:00      0    1824  1584  1584       1576      1584  140      0 [stack:992]
      c8200cc000 rw-p 00000000  00:00      0      32     0     0          0         0    0      0 
      c8200d4000 rw-p 00000000  00:00      0    1048   920   920        920       920    8      0 
      c8201da000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c8201dc000 rw-p 00000000  00:00      0     456   448   448        448       448    8      0 
      c82024e000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c820250000 rw-p 00000000  00:00      0     816   788   788        788       788   28      0 
      c82031c000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c82031e000 rw-p 00000000  00:00      0     112    96    96         96        96   16      0 
      c82033a000 rw-p 00000000  00:00      0      64    60    60         60        60    0      0 
      c82034a000 rw-p 00000000  00:00      0     728   572   572        572       572  156      0 [stack:999]
      c820400000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c820402000 rw-p 00000000  00:00      0      72    72    72         72        72    0      0 
      c820414000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c820416000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c820422000 rw-p 00000000  00:00      0      56    56    56         56        56    0      0 
      c820430000 rw-p 00000000  00:00      0     656   656   656        656       656    0      0 
      c8204d4000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c8204da000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c8204e0000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c8204e4000 rw-p 00000000  00:00      0     120   112   112        112       112    8      0 
      c820502000 rw-p 00000000  00:00      0      16     0     0          0         0    0      0 
      c820506000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c820508000 rw-p 00000000  00:00      0     368   368   368        368       368    0      0 
      c820564000 rw-p 00000000  00:00      0      16     0     0          0         0    0      0 
      c820568000 rw-p 00000000  00:00      0    3832  3768  3768       3604      3768   64      0 [stack:1046]
      c820926000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c82092c000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c82092e000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c82093a000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c82093c000 rw-p 00000000  00:00      0      16    16    16          0        16    0      0 
      c820940000 rw-p 00000000  00:00      0    1256  1256  1256       1256      1256    0      0 
      c820a7a000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c820a86000 rw-p 00000000  00:00      0     304   304   304        304       304    0      0 
      c820ad2000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c820ad6000 rw-p 00000000  00:00      0     192   120   120        120       120   72      0 
      c820b06000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c820b0c000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c820b18000 rw-p 00000000  00:00      0      32     0     0          0         0    0      0 
      c820b20000 rw-p 00000000  00:00      0    2240  2204  2204       2204      2204   36      0 
      c820d50000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c820d5c000 rw-p 00000000  00:00      0      24    20    20         20        20    4      0 
      c820d62000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c820d64000 rw-p 00000000  00:00      0     448   372   372        372       372   76      0 [stack:1155]
      c820dd4000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c820dd8000 rw-p 00000000  00:00      0     408   356   356        356       356   52      0 
      c820e3e000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c820e4a000 rw-p 00000000  00:00      0     216   216   216        216       216    0      0 
      c820e80000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c820e84000 rw-p 00000000  00:00      0    2816  2816  2816       2784      2816    0      0 
      c821144000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c821148000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c82114a000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c821156000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c821158000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c82115c000 rw-p 00000000  00:00      0     488   488   488        488       488    0      0 
      c8211d6000 rw-p 00000000  00:00      0      40    40    40         40        40    0      0 
      c8211e0000 rw-p 00000000  00:00      0     400   400   400        400       400    0      0 
      c821244000 rw-p 00000000  00:00      0      40    40    40         40        40    0      0 
      c82124e000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c821254000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c82125a000 rw-p 00000000  00:00      0     184   184   184        184       184    0      0 
      c821288000 rw-p 00000000  00:00      0      88    88    88         88        88    0      0 
      c82129e000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c8212a0000 rw-p 00000000  00:00      0      32     0     0          0         0    0      0 
      c8212a8000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c8212aa000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c8212b6000 rw-p 00000000  00:00      0     144   144   144        144       144    0      0 
      c8212da000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c8212e0000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c8212e4000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c8212f0000 rw-p 00000000  00:00      0      16    12    12         12        12    4      0 
      c8212f4000 rw-p 00000000  00:00      0      32    32    32         32        32    0      0 
      c8212fc000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c8212fe000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c821302000 rw-p 00000000  00:00      0   12600 12064 12064      11860     12064  536      0 
    7ff62cfd5000 rw-p 00000000  00:00      0     256   152   152        152       152    0      0 
    7ff62d015000 r--s 00000000  fd:01 142684 1048576  1032  1032        612         0    0      0 raft.db
    7ff66d015000 rw-p 00000000  00:00      0    1984   844   844        656       844  772      0 
    7ffc5952f000 rw-p 00000000  00:00      0     136     4     4          4         4    8      0 [stack]
    7ffc5959e000 r-xp 00000000  00:00      0       8     4     0          4         0    0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00      0       4     0     0          0         0    0      0 [vsyscall]
                                             ======= ===== ===== ========== ========= ==== ====== 
                                             1097684 37060 37056      36028     32696 1988      0 KB

jmealo commented 8 years ago

Here is the pmap output of the other node that fails regularly:

871:   consul agent -config-dir /etc/consul.d/server
         Address Perm   Offset Device  Inode    Size   Rss   Pss Referenced Anonymous Swap Locked Mapping
        00400000 r-xp 00000000  fd:01 934442    6480  1580  1529       1580         0    0      0 consul
        00a54000 r--p 00654000  fd:01 934442    6820  1536  1536       1536         0    0      0 consul
        010fd000 rw-p 00cfd000  fd:01 934442     132    68    68         68        36    0      0 consul
        0111e000 rw-p 00000000  00:00      0     184   116   116        116       116    0      0 
      c000000000 rw-p 00000000  00:00      0      32    32    32         32        32    0      0 
      c81ff16000 rw-p 00000000  00:00      0    1552  1312  1312       1272      1312  112      0 [stack:873]
      c82009a000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c8200a6000 rw-p 00000000  00:00      0     184   184   184        184       184    0      0 
      c8200d4000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c8200da000 rw-p 00000000  00:00      0     856   688   688        688       688  112      0 [stack:876]
      c8201b0000 rw-p 00000000  00:00      0      32    32    32         32        32    0      0 
      c8201b8000 rw-p 00000000  00:00      0      56    56    56         56        56    0      0 
      c8201c6000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c8201ca000 rw-p 00000000  00:00      0    1000   876   876        868       876  124      0 
      c8202c4000 rw-p 00000000  00:00      0      80    80    80         80        80    0      0 
      c8202d8000 rw-p 00000000  00:00      0     176   176   176        176       176    0      0 
      c820304000 rw-p 00000000  00:00      0      40    40    40         40        40    0      0 
      c82030e000 rw-p 00000000  00:00      0     720   700   700        664       700   20      0 [stack:1449]
      c8203c2000 rw-p 00000000  00:00      0      32    32    32         32        32    0      0 
      c8203ca000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c8203cc000 rw-p 00000000  00:00      0      40    40    40         40        40    0      0 
      c8203d6000 rw-p 00000000  00:00      0    1640  1616  1616       1476      1616   24      0 
      c820570000 rw-p 00000000  00:00      0      16    16    16         16        16    0      0 
      c820574000 rw-p 00000000  00:00      0    3080  3076  3076       2692      3076    4      0 
      c820876000 rw-p 00000000  00:00      0      32    32    32         32        32    0      0 
      c82087e000 rw-p 00000000  00:00      0     464   464   464        464       464    0      0 
      c8208f2000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c8208fe000 rw-p 00000000  00:00      0    1624  1624  1624       1076      1624    0      0 
      c820a94000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c820a9a000 rw-p 00000000  00:00      0    1040  1036  1036        464      1036    4      0 
      c820b9e000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c820ba4000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c820baa000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c820bb0000 rw-p 00000000  00:00      0     384   384   384        384       384    0      0 
      c820c10000 rw-p 00000000  00:00      0      24    24    24         24        24    0      0 
      c820c16000 rw-p 00000000  00:00      0       8     8     8          8         8    0      0 
      c820c18000 rw-p 00000000  00:00      0      40    40    40         40        40    0      0 
      c820c22000 rw-p 00000000  00:00      0     344   312   312        312       312   32      0 
      c820c78000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c820c84000 rw-p 00000000  00:00      0     936   904   904        900       904   32      0 
      c820d6e000 rw-p 00000000  00:00      0      32    32    32         32        32    0      0 
      c820d76000 rw-p 00000000  00:00      0     560   536   536        536       536   24      0 [stack:1442]
      c820e02000 rw-p 00000000  00:00      0      48    48    48         48        48    0      0 
      c820e0e000 rw-p 00000000  00:00      0   15560 15528 15528      15520     15528   32      0 
    7f764dc29000 rw-p 00000000  00:00      0     256   208   208        208       208   20      0 
    7f764dc69000 r--s 00000000  fd:01 142683 1048576    56    56         56         0    0      0 raft.db
    7f768dc69000 rw-p 00000000  00:00      0    1984  1164  1164        716      1164  480      0 
    7fff632ac000 rw-p 00000000  00:00      0     136     4     4          4         4   16      0 [stack]
    7fff633c7000 r-xp 00000000  00:00      0       8     4     0          4         0    0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00      0       4     0     0          0         0    0      0 [vsyscall]
                                             ======= ===== ===== ========== ========= ==== ====== 
                                             1135456 40740 40607      37868     37404 3496      0 KB

Here is the pmap output of a node that does not exhibit the failure behavior:

22760:   consul agent -config-dir /etc/consul.d/server
         Address Perm   Offset Device   Inode    Size    Rss    Pss Referenced Anonymous Swap Locked Mapping
        00400000 r-xp 00000000  fd:01  922600    6480   4212   4212       4212         0    0      0 consul
        00a54000 r--p 00654000  fd:01  922600    6820   3772   3772       3772         0    0      0 consul
        010fd000 rw-p 00cfd000  fd:01  922600     132     96     96         96        36    0      0 consul
        0111e000 rw-p 00000000  00:00       0     184    116    116        116       116    0      0 
      c000000000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c81fec8000 rw-p 00000000  00:00       0    2312   2152   2152       2148      2152    0      0 [stack:22761]
      c82010a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82010c000 rw-p 00000000  00:00       0     176    176    176        176       176    0      0 [stack:22769]
      c820138000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82013a000 rw-p 00000000  00:00       0     304    304    304        304       304    0      0 
      c820186000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820188000 rw-p 00000000  00:00       0      80     24     24         24        24    0      0 
      c82019c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82019e000 rw-p 00000000  00:00       0     104     76     76         76        76    0      0 
      c8201b8000 rw-p 00000000  00:00       0      88     88     88         88        88    0      0 
      c8201ce000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8201d0000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8201d4000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c8201dc000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8201e0000 rw-p 00000000  00:00       0     304    304    304        304       304    0      0 
      c82022c000 rw-p 00000000  00:00       0      80     80     80         80        80    0      0 
      c820240000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820244000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820246000 rw-p 00000000  00:00       0     232    232    232        232       232    0      0 
      c820280000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820288000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c820298000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82029c000 rw-p 00000000  00:00       0     256    228    228        228       228    0      0 [stack:22766]
      c8202dc000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202de000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202e0000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8202e4000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 [stack:22881]
      c8202ec000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8202f0000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c820302000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820304000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820306000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820308000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82030a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82030c000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820312000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820320000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820324000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820326000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c820336000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82033a000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82033e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820340000 rw-p 00000000  00:00       0     136    136    136        136       136    0      0 
      c820362000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820370000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c82037c000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820386000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c820396000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820398000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c8203a8000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8203aa000 rw-p 00000000  00:00       0     120    120    120        120       120    0      0 
      c8203c8000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c8203d4000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c8203de000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8203e0000 rw-p 00000000  00:00       0     136    136    136        136       136    0      0 
      c820402000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820404000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820406000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820408000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820416000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c82041e000 rw-p 00000000  00:00       0     200    200    200        200       200    0      0 
      c820450000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820458000 rw-p 00000000  00:00       0     136    136    136        136       136    0      0 
      c82047a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82047c000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820484000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820486000 rw-p 00000000  00:00       0     152    152    152        152       152    0      0 
      c8204ac000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8204ae000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c8204ba000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8204bc000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8204be000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c8204c8000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c8204d0000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c8204e2000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8204e6000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c8204f4000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8204f6000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8204fa000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c820506000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c820516000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c82051c000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820522000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820524000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c82052c000 rw-p 00000000  00:00       0      88     88     88         88        88    0      0 
      c820542000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820548000 rw-p 00000000  00:00       0     248    248    248        248       248    0      0 
      c820586000 rw-p 00000000  00:00       0      80     80     80         80        80    0      0 
      c82059a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82059c000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8205a0000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8205a4000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c8205b2000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205b4000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c8205be000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205c0000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205c2000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205c4000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205c6000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205c8000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8205cc000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205ce000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c8205d4000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c8205e6000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205e8000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205ea000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205ec000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c8205f4000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205f6000 rw-p 00000000  00:00       0      80     80     80         80        80    0      0 
      c82060a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82060c000 rw-p 00000000  00:00       0    2992   2964   2964       2964      2964    0      0 
      c8208f8000 rw-p 00000000  00:00       0     232    232    232        232       232    0      0 
      c820932000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820934000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820936000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82093a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82093c000 rw-p 00000000  00:00       0     160    160    160        160       160    0      0 
      c820964000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820972000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c82097a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82097c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82097e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820980000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820984000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820992000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820996000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c82099e000 rw-p 00000000  00:00       0     776      0      0          0         0    0      0 
      c820a60000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c820a72000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820a76000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820a84000 rw-p 00000000  00:00       0      80     80     80         80        80    0      0 
      c820a98000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820a9e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820aa0000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820aa4000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 [stack:6045]
      c820aa8000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820ab0000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820ab2000 rw-p 00000000  00:00       0    1152   1152   1152       1152      1152    0      0 
      c820bd2000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820bda000 rw-p 00000000  00:00       0     288    288    288        288       288    0      0 
      c820c22000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820c30000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820c34000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820c42000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820c46000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820c54000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c820c60000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820c64000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c820c70000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c72000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820c78000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c7a000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820c88000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c8a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c8c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c8e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c90000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820c9e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820ca0000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c820cac000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820cae000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820cbc000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c820cc8000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820ccc000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c820cdc000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820cde000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820ce2000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820ce6000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820ce8000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 [stack:3431]
      c820cf2000 rw-p 00000000  00:00       0      96     96     96         96        96    0      0 
      c820d0a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820d0c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820d0e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820d10000 rw-p 00000000  00:00       0      80     80     80         80        80    0      0 
      c820d24000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820d26000 rw-p 00000000  00:00       0      96     96     96         96        96    0      0 
      c820d3e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820d40000 rw-p 00000000  00:00       0      96     96     96         96        96    0      0 
      c820d58000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820d5c000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c820d68000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820d6a000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820d70000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820d74000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820d7a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820d7c000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820d80000 rw-p 00000000  00:00       0    4696   4696   4696       4696      4696    0      0 
      c821216000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821218000 rw-p 00000000  00:00       0    1808   1808   1808       1808      1808    0      0 
      c8213dc000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8213de000 rw-p 00000000  00:00       0     696    696    696        696       696    0      0 
      c82148c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82148e000 rw-p 00000000  00:00       0    4800   4800   4800       4800      4800    0      0 
      c82193e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821940000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821946000 rw-p 00000000  00:00       0    1088   1088   1088       1088      1088    0      0 
      c821a56000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821a5c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821a5e000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821a64000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c821a70000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821a76000 rw-p 00000000  00:00       0    1120   1120   1120       1120      1120    0      0 
      c821b8e000 rw-p 00000000  00:00       0    1000   1000   1000       1000      1000    0      0 
      c821c88000 rw-p 00000000  00:00       0     456    456    456        456       456    0      0 
      c821cfa000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c821cfe000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821d00000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c821d12000 rw-p 00000000  00:00       0     160    160    160        160       160    0      0 
      c821d3a000 rw-p 00000000  00:00       0      96     96     96         96        96    0      0 
      c821d52000 rw-p 00000000  00:00       0    9720   9720   9720       9720      9720    0      0 
    7f4a4ba06000 rw-p 00000000  00:00       0    1024    564    564        564       564    0      0 
    7f4a4bb06000 r--s 00000000  fd:01 1310787 1048576  90784  90784      90784         0    0      0 raft.db
    7f4a8bb06000 rw-p 00000000  00:00       0    2240   1948   1948       1948      1948    0      0 
    7ffc19f4f000 rw-p 00000000  00:00       0     136     16     16         16        16    0      0 [stack]
    7ffc19fed000 r-xp 00000000  00:00       0       8      4      0          4         0    0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00       0       4      0      0          0         0    0      0 [vsyscall]
                                              ======= ====== ====== ========== ========= ==== ====== 
                                              1106636 141468 141464     141464     42636    0      0 KB

And another node that does not have this behavior:

root@redacted:~# pmap -X $(pidof consul)
22211:   consul agent -config-dir /etc/consul.d/server
         Address Perm   Offset Device   Inode    Size    Rss    Pss Referenced Anonymous Swap Locked Mapping
        00400000 r-xp 00000000  fd:01  920234    6480   2820   2820       2820         0    0      0 consul
        00a54000 r--p 00654000  fd:01  920234    6820   2504   2504       2504         0    0      0 consul
        010fd000 rw-p 00cfd000  fd:01  920234     132     84     84         84        36    0      0 consul
        0111e000 rw-p 00000000  00:00       0     184    116    116        116       116    0      0 
      c000000000 rw-p 00000000  00:00       0      36     36     36         36        36    0      0 
      c81fee6000 rw-p 00000000  00:00       0    1648   1548   1548       1548      1548    0      0 [stack:22212]
      c820082000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820084000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c82008a000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82008e000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820098000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c8200aa000 rw-p 00000000  00:00       0     240    212    212        212       212    0      0 
      c8200e6000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c8200ec000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c8200f2000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8200f4000 rw-p 00000000  00:00       0     328    272    272        272       272    0      0 
      c820146000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820154000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820158000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820162000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c82016a000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820170000 rw-p 00000000  00:00       0      88     88     88         88        88    0      0 
      c820186000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82018a000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820194000 rw-p 00000000  00:00       0     112    112    112        112       112    0      0 
      c8201b0000 rw-p 00000000  00:00       0      88     88     88         88        88    0      0 
      c8201c6000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c8201d0000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8201d4000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c8201e4000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c8201f4000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8201f6000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8201f8000 rw-p 00000000  00:00       0     120     60     60         60        60    0      0 
      c820216000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820218000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82021c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82021e000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820228000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82022a000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820230000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820234000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820238000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82023a000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820240000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820242000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c82024a000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82024e000 rw-p 00000000  00:00       0      96     96     96         96        96    0      0 
      c820266000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820268000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 [stack:22229]
      c820272000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820276000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82027a000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82027e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820280000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820286000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c82028e000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c82029a000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82029e000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c8202a8000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202aa000 rw-p 00000000  00:00       0     176    176    176        176       176    0      0 [stack:22217]
      c8202d6000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202d8000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202da000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202dc000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8202e0000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202e2000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202e4000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8202e8000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202ea000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c8202f0000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202f2000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c8202f8000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8202fa000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c820308000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820310000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c82031c000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820324000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820326000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c820338000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82033a000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c82034a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82034c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82034e000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820358000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c820364000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c82036e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820370000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c82037c000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820386000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820388000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820392000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820398000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82039c000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c8203a6000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8203aa000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8203ac000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8203ae000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c8203ba000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8203bc000 rw-p 00000000  00:00       0     168    168    168        168       168    0      0 
      c8203e6000 rw-p 00000000  00:00       0     152    152    152        152       152    0      0 
      c82040c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82040e000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c82041c000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820422000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820424000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c82042a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82042c000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c82043e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820440000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820442000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820444000 rw-p 00000000  00:00       0     832    832    832        832       832    0      0 
      c820514000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820516000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c82051c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82051e000 rw-p 00000000  00:00       0     128    128    128        128       128    0      0 
      c82053e000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820544000 rw-p 00000000  00:00       0      96     96     96         96        96    0      0 
      c82055c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82055e000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820562000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820568000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82056c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82056e000 rw-p 00000000  00:00       0     112    112    112        112       112    0      0 
      c82058a000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820590000 rw-p 00000000  00:00       0     216    216    216        216       216    0      0 
      c8205c6000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c8205ce000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c8205d4000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205d6000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 [stack:32755]
      c8205e6000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205e8000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8205ea000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8205ee000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c8205f8000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820600000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820604000 rw-p 00000000  00:00       0     112    112    112        112       112    0      0 
      c820620000 rw-p 00000000  00:00       0      96     96     96         96        96    0      0 
      c820638000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c82064a000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820652000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c82065a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82065c000 rw-p 00000000  00:00       0     200    200    200        200       200    0      0 
      c82068e000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820692000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820694000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c82069c000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c8206a6000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8206a8000 rw-p 00000000  00:00       0     144    144    144        144       144    0      0 
      c8206cc000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c8206dc000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8206de000 rw-p 00000000  00:00       0    1016   1016   1016       1016      1016    0      0 
      c8207dc000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8207de000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c8207ec000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c8207f0000 rw-p 00000000  00:00       0     320    320    320        320       320    0      0 
      c820840000 rw-p 00000000  00:00       0      80     80     80         80        80    0      0 
      c820854000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820856000 rw-p 00000000  00:00       0     104    104    104        104       104    0      0 
      c820870000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820872000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c820882000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820886000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820888000 rw-p 00000000  00:00       0      88     88     88         88        88    0      0 
      c82089e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8208a0000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c8208a8000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c8208b0000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c8208c0000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c8208d0000 rw-p 00000000  00:00       0    2408   2408   2408       2408      2408    0      0 
      c820b2a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820b2c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820b2e000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820b38000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820b3a000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820b3e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820b40000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820b46000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820b4a000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820b52000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820b54000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820b5e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820b60000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820b68000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820b6a000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820b74000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820b7a000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820b7e000 rw-p 00000000  00:00       0     184    184    184        184       184    0      0 
      c820bac000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820bb0000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820bb2000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820bbc000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820bbe000 rw-p 00000000  00:00       0     176    176    176        176       176    0      0 
      c820bea000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820bec000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820bf0000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820bf2000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820bfa000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820bfc000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c820c02000 rw-p 00000000  00:00       0     112    112    112        112       112    0      0 
      c820c1e000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c820c28000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c2a000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820c32000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c820c44000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c46000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c48000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c820c4a000 rw-p 00000000  00:00       0    2880   2880   2880       2880      2880    0      0 
      c820f1a000 rw-p 00000000  00:00       0     632    632    632        632       632    0      0 
      c820fb8000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c820fbc000 rw-p 00000000  00:00       0     224    224    224        224       224    0      0 
      c820ff4000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c820ffc000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c82100a000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c821012000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821018000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c82101e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821020000 rw-p 00000000  00:00       0      96     96     96         96        96    0      0 
      c821038000 rw-p 00000000  00:00       0      72     72     72         72        72    0      0 
      c82104a000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82104e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821050000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c82105e000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c82106c000 rw-p 00000000  00:00       0      80     80     80         80        80    0      0 [stack:24413]
      c821080000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821086000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82108a000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c821098000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c82109e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8210a0000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c8210a8000 rw-p 00000000  00:00       0     216    216    216        216       216    0      0 
      c8210de000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8210e0000 rw-p 00000000  00:00       0      64     64     64         64        64    0      0 
      c8210f0000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c8210fc000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821102000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821104000 rw-p 00000000  00:00       0     360    360    360        360       360    0      0 
      c82115e000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821164000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821166000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821168000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82116a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82116c000 rw-p 00000000  00:00       0     464    464    464        464       464    0      0 
      c8211e0000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8211e2000 rw-p 00000000  00:00       0      48     48     48         48        48    0      0 
      c8211ee000 rw-p 00000000  00:00       0      56     56     56         56        56    0      0 
      c8211fc000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c8211fe000 rw-p 00000000  00:00       0     224    224    224        224       224    0      0 
      c821236000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821238000 rw-p 00000000  00:00       0     120    120    120        120       120    0      0 
      c821256000 rw-p 00000000  00:00       0      16     16     16         16        16    0      0 
      c82125a000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82125c000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c82125e000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821260000 rw-p 00000000  00:00       0    3336   3336   3336       3336      3336    0      0 
      c8215a2000 rw-p 00000000  00:00       0    8176   8176   8176       8176      8176    0      0 
      c821d9e000 rw-p 00000000  00:00       0    1096   1096   1096       1096      1096    0      0 
      c821eb0000 rw-p 00000000  00:00       0       8      8      8          8         8    0      0 
      c821eb2000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821eb8000 rw-p 00000000  00:00       0      24     24     24         24        24    0      0 
      c821ebe000 rw-p 00000000  00:00       0      32     32     32         32        32    0      0 
      c821ec6000 rw-p 00000000  00:00       0      40     40     40         40        40    0      0 
      c821ed0000 rw-p 00000000  00:00       0     208    208    208        208       208    0      0 
      c821f04000 rw-p 00000000  00:00       0    4144   4144   4144       4144      4144    0      0 
    7f53188ca000 rw-p 00000000  00:00       0     768    720    720        720       720    0      0 
    7f531898a000 r--s 00000000  fd:01 1182250 1048576  64804  64804      64652         0    0      0 raft.db
    7f535898a000 rw-p 00000000  00:00       0    1984   1692   1692       1692      1692    0      0 
    7ffde2788000 rw-p 00000000  00:00       0     136     20     20         20        20    0      0 [stack]
    7ffde27ed000 r-xp 00000000  00:00       0       8      4      0          4         0    0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00       0       4      0      0          0         0    0      0 [vsyscall]
                                              ======= ====== ====== ========== ========= ==== ====== 
                                              1102160 109588 109584     109436     39408    0      0 KB

jmealo commented 8 years ago

@slackpad kill -s USR1 $(pidof consul) doesn't seem to work for me when syslog is enabled in the consul server config file. Do I need to run the process in the foreground or pipe stderr to a file to dump telemetry?

slackpad commented 8 years ago

@jmealo yes for telemetry it won't go to the syslog writer, it'll be written to your Consul server's stdout so you'll need to grab that wherever it is being logged.

jmealo commented 8 years ago

@slackpad: It's been 1 day 20 minutes since I disabled consul on my cluster and we've experienced no memory issues. Prior to disabling consul I had to reboot the servers every 6 hours to clear memory. I'll rest easier if we let it go another couple of days, but, this configuration worked for 2-3 months without Consul and removing it seems to immediately solve the problem, so I can say with some certainty that Consul is the issue.

I don't know why only the load balancers have this issue, the only difference in their configuration is the allowed number of open files is significantly higher, which could possibly be allowing something to leak handles.

To me the pmap output makes it clear that raft.db is using a significant amount of memory. Perhaps log compaction is leaking memory?

If it helps at all this happened in 0.5.x and 0.6.x so the large changes in 0.6.x probably don't have anything to do with it.

slackpad commented 8 years ago

@jmealo do you have any updates on this one? I think it's unusual in general to run application stuff like load balancers on your Consul server nodes, typically folks want to keep them isolated. In the pmap outputs above the RSS values for Consul look to be ~40 MB which is pretty reasonable.

jmealo commented 8 years ago

@slackpad Thank you for following up. As excited as we were about consul we ended up reimplementing everything we needed from it in Node in about a day and have had zero infrastructure issues since we got rid of Consul.

My only update is that Consul was the cause of the OOM issues.

I guess it's safe to close this ticket; however, the problem exists and causes catastrophic failures that could take every node in a cluster down one-by-one within a few minutes of each other if consul is started at the same time on each node where this condition is present. This would obviously be bad but also likely require the user to redo the bootstrap process.

That might be a good thing, because if the cluster goes down and comes back up and consul isn't working, it won't take down the cluster again.

We sunk a couple of weeks of troubleshooting and hours of downtime in production into this and I hope that nobody else goes down that rabbit hole.

If you think of it whenever the problem is finally solved or encountered on your end please ping me :-)

mitchellh commented 8 years ago

I'm glad you found a solution @jmealo. If you ever get around to it we'd love to look into this further with you but if you're happy with your current solution its best to leave it as-is.

We work personally (1-on-1) with a handful of the largest websites that exist that run gigantic Consul clusters and in the past we have found memory leaks we've fixed straight away (none recently to my knowledge). These clusters generally run with a long uptime and their memory usage is stable. They use every feature of Consul (at least prior to 0.6, folks are still adopting 0.6 features) and have never experienced memory issues.

None of this means that you didn't have a real issue, but I'm confident that in the general case Consul is extremely stable and memory is well under control. There may have been something unique about your setup we missed through the back and forth here, though. I think @slackpad overall was correct and I mirror his view that all your debug output seems to point to Consul using a very reasonable amount of memory. So we must be missing some data somewhere.

Good luck and let us know if we can help in the future.

(EDIT: sorry "james" in GitHub for pinging you with this, I meant @slackpad)

jmealo commented 8 years ago

@mitchellh I'd really like to see this fixed. My node solution simply parses my consul health checks and runs them using Node instead of Consul. The health checks are unchanged except for one that checked a socket.io server by requesting /socket.io.js which is 170kb uncompressed and seemed the likely cause of the issue.

I can say with high confidence that the health checks were not the issue as they are running from Node without issue. As part of our consul troubleshooting, the socket.io test was disabled without relief.

The issue at hand here is consul/go-raft/go was causing a memory leak that:

Consul's telemetry doesn't know about
Every "normal" way of checking memory and shared memory allocation in Linux provided no insight into the cause of the issue

My primary concern is that another enduser will go absolutely bonkers trying to figure this out. I myself was convinced that consul was not the issue based on our troubleshooting. I went to DigitalOcean, KVM, linux-kernel NGINX, OpenResty, consul trying to figure out the memory usage issue to no avail.

If I can get a box setup and recreate the conditions can you delegate engineering time to work on this?I'm not sure if I can reproduce it without the other 7 nodes in the cluster but it might be worth a try.

The stack that caused this will be open sourced in the not so distant future (a year tops). I may be able to provide bootstrap scripts for an entire cluster identical to this one at some point if you want to test it internally.

Thanks, Jeff

EDIT: Additional info on largest health check response and wording.

slackpad commented 8 years ago

@jmealo I'll go ahead and kick this one back open. If you can get a repro environment that won't disrupt your operations and that we can poke at I can give you some cycles to try to see what's going on. Appreciate you offering to set that up!

If you had a health check that was logging on the order of megabytes and churning it could cause some bad GC behavior possibly, depending on how often it was running. That would be something to take a look at.

With regards to a richer HTTP check we've gotten requests for that but generally avoided adding more features in favor of people just scripting their own with curl or wget if they need more features. There are so many possible options and we want to keep the Consul core as simple as we can, unless a new feature is a huge win for a lot of different use cases.

jmealo commented 8 years ago

@slackpad @mitchellh From the looks of it this was a kernel leak caused by non-process code (which is why the memory could not be attributed to a process).

That being said, using OpenResty and consul together can reproduce the issue. Using either in isolation does not.

The issue did not happen when running vanilla NGINX with consul. The only difference is that OpenResty is serving 1-2k connections and NGINX was in the low hundreds. Since both the primary and failover load balancers would both fail after a set amount of time regardless of connection or load, I think it's safe to say that the network traffic has nothing to do with it.

In the interest of saving everyone time (unless someone wants to learn how to diagnose kernel memory leaks in KVM on a cloud provider) I'd be willing to give it another go when Ubuntu 16 LTS is released.

Does that sound like a good plan?

slackpad commented 8 years ago

Hi @jmealo thanks for the update - that sounds like a good plan!

slackpad commented 7 years ago

Haven't heard more about this one so we will close it out for now, but please let us know if you have more information or need help.

hashicorp / consul

Kernel Memory Leak: possible conflict between LuaJIT or OpenResty and the consul server agent #1476

This is not related to #1060.