server failure: scoretility and steeplechasers.org

[louking]

orphaned connections in close_wait state

• https://unix.stackexchange.com/questions/10106/orphaned-connections-in-close-wait-state
  • http://unix.derkeiler.com/Mailing-Lists/SunManagers/2006-01/msg00367.html
• https://serverfault.com/questions/160558/how-to-not-get-so-many-apache-close-wait-connections
• http://www.linuxforums.org/forum/applications/192103-apache-close_wait-connections.html
• https://stackoverflow.com/questions/15912370/how-do-i-remove-a-close-wait-socket-connection

logging ip traffic

• https://www.tecmint.com/20-netstat-commands-for-linux-network-management/
• http://www.binarytides.com/linux-commands-monitor-network/
• https://www.tecmint.com/dstat-monitor-linux-server-performance-process-memory-network/

see

• https://serverfault.com/questions/772765/apache-spawning-enormous-number-of-httpd-processes
• https://unix.stackexchange.com/questions/170671/memory-consumption-fail2ban-centos
• https://stackoverflow.com/questions/6481760/reduce-rabbitmq-memory-usage
• https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-centos-7
• https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/3/html/System_Administration_Guide/s1-swap-adding.html
• DOS / brute force
  • https://www.digitalocean.com/community/tutorials/how-to-protect-against-dos-and-ddos-with-mod_evasive-for-apache-on-centos-7
  • https://securityintelligence.com/defending-against-apache-web-server-ddos-attacks/
  • https://www.plesk.com/blog/product-technology/protecting-your-wordpress-installations-with-fail2ban
  • https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-centos-7

possibly for recovery

• when low swap space detected
  • restart apache
  • restart mariadb
  • start tcpdump or ssldump - http://packetpushers.net/using-ssldump-decode-ssltls-packets/
  • check https://serverfault.com/questions/56526/whats-the-best-way-of-setting-up-a-cron-job-to-check-that-a-long-running-proces#

mariadb.log:

170804 15:44:42 mysqld_safe Number of processes running now: 0
170804 15:44:42 mysqld_safe mysqld restarted
170804 15:44:42 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 15328 ...
170804 15:44:43 InnoDB: The InnoDB memory heap is disabled
170804 15:44:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170804 15:44:43 InnoDB: Compressed tables use zlib 1.2.7
170804 15:44:43 InnoDB: Using Linux native AIO
170804 15:44:43 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
170804 15:44:43 InnoDB: Completed initialization of buffer pool
170804 15:44:43 InnoDB: Fatal error: cannot allocate memory for the buffer pool
170804 15:44:43 [ERROR] Plugin 'InnoDB' init function returned error.
170804 15:44:43 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
170804 15:44:43 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
170804 15:44:43 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
170804 15:44:43 [Note] Plugin 'FEEDBACK' is disabled.
170804 15:44:43 [ERROR] Unknown/unsupported storage engine: InnoDB
170804 15:44:43 [ERROR] Aborting

messages:

Aug  4 15:44:40 loutilities kernel: celery invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug  4 15:44:40 loutilities kernel: celery cpuset=/ mems_allowed=0
Aug  4 15:44:40 loutilities kernel: CPU: 0 PID: 2009 Comm: celery Not tainted 3.10.0-327.18.2.el7.x86_64 #1
Aug  4 15:44:40 loutilities kernel: Hardware name: DigitalOcean Droplet, BIOS 20161103 11/03/2016
Aug  4 15:44:40 loutilities kernel: ffff880078e2dc00 0000000027f319cd ffff880078cb39c8 ffffffff81635a0c
Aug  4 15:44:40 loutilities kernel: ffff880078cb3a58 ffffffff816309ac ffff880078e24bb0 ffff880078e24bc8
Aug  4 15:44:40 loutilities kernel: 0000000000000206 ffff880078e2dc00 ffff880078cb3a40 ffffffff81128b1f
Aug  4 15:44:40 loutilities kernel: Call Trace:
Aug  4 15:44:40 loutilities kernel: [<ffffffff81635a0c>] dump_stack+0x19/0x1b
Aug  4 15:44:40 loutilities kernel: [<ffffffff816309ac>] dump_header+0x8e/0x214
Aug  4 15:44:40 loutilities kernel: [<ffffffff81128b1f>] ? delayacct_end+0x8f/0xb0
Aug  4 15:44:40 loutilities kernel: [<ffffffff8116d0be>] oom_kill_process+0x24e/0x3b0
Aug  4 15:44:40 loutilities kernel: [<ffffffff8116cc26>] ? find_lock_task_mm+0x56/0xc0
Aug  4 15:44:40 loutilities kernel: [<ffffffff81088dae>] ? has_capability_noaudit+0x1e/0x30
Aug  4 15:44:40 loutilities kernel: [<ffffffff8116d8e6>] out_of_memory+0x4b6/0x4f0
Aug  4 15:44:40 loutilities kernel: [<ffffffff81173ac5>] __alloc_pages_nodemask+0xa95/0xb90
Aug  4 15:44:40 loutilities kernel: [<ffffffff811b46b9>] alloc_pages_current+0xa9/0x170
Aug  4 15:44:40 loutilities kernel: [<ffffffff811692c7>] __page_cache_alloc+0x97/0xc0
Aug  4 15:44:40 loutilities kernel: [<ffffffff8116bb30>] filemap_fault+0x170/0x410
Aug  4 15:44:40 loutilities kernel: [<ffffffff81192dfe>] __do_fault+0x7e/0x510
Aug  4 15:44:40 loutilities kernel: [<ffffffff81197358>] handle_mm_fault+0x5b8/0xf50
Aug  4 15:44:40 loutilities kernel: [<ffffffff811e3d6f>] ? cp_new_stat+0x14f/0x180
Aug  4 15:44:40 loutilities kernel: [<ffffffff816416c0>] __do_page_fault+0x150/0x450
Aug  4 15:44:40 loutilities kernel: [<ffffffff81641a83>] trace_do_page_fault+0x43/0x110
Aug  4 15:44:40 loutilities kernel: [<ffffffff81641179>] do_async_page_fault+0x29/0xe0
Aug  4 15:44:40 loutilities kernel: [<ffffffff8163dc78>] async_page_fault+0x28/0x30
Aug  4 15:44:40 loutilities kernel: Mem-Info:
Aug  4 15:44:40 loutilities kernel: Node 0 DMA per-cpu:
Aug  4 15:44:40 loutilities kernel: CPU    0: hi:    0, btch:   1 usd:   0
Aug  4 15:44:40 loutilities kernel: CPU    1: hi:    0, btch:   1 usd:   0
Aug  4 15:44:40 loutilities kernel: Node 0 DMA32 per-cpu:
Aug  4 15:44:40 loutilities kernel: CPU    0: hi:  186, btch:  31 usd:   0
Aug  4 15:44:41 loutilities kernel: CPU    1: hi:  186, btch:  31 usd: 157
Aug  4 15:44:41 loutilities kernel: active_anon:342153 inactive_anon:114617 isolated_anon:0#012 active_file:1411 inactive_file:1387 isolated_file:32#012 unevictable:0 dirty:2 writeback:0 unstable:0#012 free:14217 slab_reclaimable:4605 slab_unreclaimable:7935#012 mapped:681 shmem:162 pagetables:15509 bounce:0#012 free_cma:0
Aug  4 15:44:41 loutilities kernel: Node 0 DMA free:8260kB min:348kB low:432kB high:520kB active_anon:3036kB inactive_anon:3136kB active_file:8kB inactive_file:80kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:4kB slab_reclaimable:88kB slab_unreclaimable:236kB kernel_stack:224kB pagetables:332kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:44 all_unreclaimable? no
Aug  4 15:44:41 loutilities kernel: lowmem_reserve[]: 0 1983 1983 1983
Aug  4 15:44:41 loutilities kernel: Node 0 DMA32 free:48704kB min:44704kB low:55880kB high:67056kB active_anon:1365576kB inactive_anon:455332kB active_file:5636kB inactive_file:5468kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080760kB managed:2033080kB mlocked:0kB dirty:8kB writeback:0kB mapped:2600kB shmem:644kB slab_reclaimable:18332kB slab_unreclaimable:31504kB kernel_stack:5504kB pagetables:61704kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug  4 15:44:41 loutilities kernel: lowmem_reserve[]: 0 0 0 0
Aug  4 15:44:41 loutilities kernel: Node 0 DMA: 3*4kB (UE) 69*8kB (EM) 45*16kB (UEM) 7*32kB (UEM) 2*64kB (EM) 2*128kB (U) 3*256kB (UEM) 3*512kB (UEM) 2*1024kB (UE) 1*2048kB (R) 0*4096kB = 8292kB
Aug  4 15:44:41 loutilities kernel: Node 0 DMA32: 1677*4kB (UEM) 634*8kB (UEM) 814*16kB (UEM) 549*32kB (UEM) 48*64kB (UEM) 8*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 48772kB
Aug  4 15:44:41 loutilities kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Aug  4 15:44:41 loutilities kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug  4 15:44:41 loutilities kernel: 4292 total pagecache pages
Aug  4 15:44:41 loutilities kernel: 1327 pages in swap cache
Aug  4 15:44:41 loutilities kernel: Swap cache stats: add 5705792, delete 5704465, find 3123833/3587373
Aug  4 15:44:41 loutilities kernel: Free swap  = 0kB
Aug  4 15:44:41 loutilities kernel: Total swap = 1048572kB
Aug  4 15:44:41 loutilities kernel: 524188 pages RAM
Aug  4 15:44:41 loutilities kernel: 0 pages HighMem/MovableOnly
Aug  4 15:44:41 loutilities kernel: 11941 pages reserved
Aug  4 15:44:41 loutilities kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Aug  4 15:44:41 loutilities kernel: [  342]     0   342     9301      125      22       61             0 systemd-journal
Aug  4 15:44:41 loutilities kernel: [  370]     0   370    10877        1      22      163         -1000 systemd-udevd
Aug  4 15:44:41 loutilities kernel: [  380]     0   380    13854        0      26      113         -1000 auditd
Aug  4 15:44:41 loutilities kernel: [  423]    81   423     6136       53      17       65          -900 dbus-daemon
Aug  4 15:44:41 loutilities kernel: [  456]   999   456   132068        0      54     2045             0 polkitd
Aug  4 15:44:41 loutilities kernel: [  457]     0   457     6048       27      16       50             0 systemd-logind
Aug  4 15:44:41 loutilities kernel: [  461]     0   461     4826       25      14       37             0 irqbalance
Aug  4 15:44:41 loutilities kernel: [  474]    38   474    11161       32      26      142             0 ntpd
Aug  4 15:44:41 loutilities kernel: [  475]     0   475    31555       16      17      140             0 crond
Aug  4 15:44:41 loutilities kernel: [  489]     0   489    27509        1      10       31             0 agetty
Aug  4 15:44:41 loutilities kernel: [  495]     0   495    82880        2      81     6136             0 firewalld
Aug  4 15:44:41 loutilities kernel: [  806]     0   806    54944       32      37      151             0 rsyslogd
Aug  4 15:44:41 loutilities kernel: [  807]   996   807   718474    10193     117    13316             0 beam.smp
Aug  4 15:44:41 loutilities kernel: [  811]     0   811    26370        0      53      246         -1000 sshd
Aug  4 15:44:41 loutilities kernel: [  812]     0   812   138316       90      89     2597             0 tuned
Aug  4 15:44:41 loutilities kernel: [  813]     0   813   104340      395     196     1501             0 httpd
Aug  4 15:44:41 loutilities kernel: [  932]    27   932    28315        0      11       77             0 mysqld_safe
Aug  4 15:44:41 loutilities kernel: [ 1194]     0  1194    69137      291     131      752             0 httpd
Aug  4 15:44:41 loutilities kernel: [ 1226]     0  1226   360533     1215     124     5095             0 fail2ban-server
Aug  4 15:44:41 loutilities kernel: [ 1323]  1000  1323   238779      855     275    15978             0 httpd
Aug  4 15:44:41 loutilities kernel: [ 1324]  1000  1324   149923      859     196     1950             0 httpd
Aug  4 15:44:41 loutilities kernel: [ 1325]  1000  1325   149923      859     196     1950             0 httpd
Aug  4 15:44:41 loutilities kernel: [ 1326]  1000  1326   222525     4016     274    13155             0 httpd
Aug  4 15:44:41 loutilities kernel: [ 1327]  1003  1327   141727      868     192     1941             0 httpd
Aug  4 15:44:41 loutilities kernel: [ 1330]  1003  1330   145825      862     194     1947             0 httpd
Aug  4 15:44:41 loutilities kernel: [ 1573]     0  1573    55377      416      59     2966             0 supervisord
Aug  4 15:44:41 loutilities kernel: [ 1591]   996  1591     2893       18      11       22             0 epmd
Aug  4 15:44:41 loutilities kernel: [ 1635]     0  1635    22259       20      42      240             0 master
Aug  4 15:44:41 loutilities kernel: [ 1657]    89  1657    22333       24      43      246             0 qmgr
Aug  4 15:44:41 loutilities kernel: [ 2006]  1000  2006    72409        1      97    13369             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2007]  1000  2007    72738     2001      95    11712             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2008]  1000  2008    72738     1497      97    12299             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2009]  1000  2009    72752     1491      95    12267             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2010]  1000  2010    72748     1964      95    11776             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2550]   996  2550     1075        9       8       13             0 erl_child_setup
Aug  4 15:44:41 loutilities kernel: [ 2715]  1000  2715    97007      270     109    16217             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2716]  1000  2716    72455        0      92    13453             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2717]  1000  2717    72453      386      91    13071             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2718]  1000  2718    72456      346      91    13112             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2719]  1000  2719    72176        0      91    13173             0 celery
Aug  4 15:44:41 loutilities kernel: [ 2765]   996  2765     2884        6      11       23             0 inet_gethost
Aug  4 15:44:41 loutilities kernel: [ 2766]   996  2766     4471       28      13       15             0 inet_gethost
Aug  4 15:44:41 loutilities kernel: [26718]    27 26718   413004     7554     152    25596             0 mysqld
Aug  4 15:44:41 loutilities kernel: [ 2941]     0  2941    36943        6      74      318             0 sshd
Aug  4 15:44:41 loutilities kernel: [ 2945]  1002  2945    36943       33      72      308             0 sshd
Aug  4 15:44:41 loutilities kernel: [ 2946]  1002  2946    28869        1      12      127             0 bash
Aug  4 15:44:41 loutilities kernel: [13385]    89 13385    22285       13      43      239             0 pickup
Aug  4 15:44:41 loutilities kernel: [13979]  1000 13979   238796    13994     278     4807             0 httpd
Aug  4 15:44:41 loutilities kernel: [14880]    48 14880   128624      829     241    25455             0 httpd
Aug  4 15:44:41 loutilities kernel: [14881]    48 14881   129293      859     243    26302             0 httpd
Aug  4 15:44:41 loutilities kernel: [14905]    48 14905   130901    27108     246     1392             0 httpd
Aug  4 15:44:41 loutilities kernel: [14906]    48 14906   148961    23778     253     4111             0 httpd
Aug  4 15:44:41 loutilities kernel: [14970]    48 14970   150631      810     256    28532             0 httpd
Aug  4 15:44:41 loutilities kernel: [15078]    48 15078   130730    22989     245     5441             0 httpd
Aug  4 15:44:41 loutilities kernel: [15172]    48 15172   104957     1296     192     1176             0 httpd
Aug  4 15:44:41 loutilities kernel: [15181]    89 15181    22286      251      42        0             0 trivial-rewrite
Aug  4 15:44:41 loutilities kernel: [15182]    89 15182    22337      259      43        0             0 smtp
Aug  4 15:44:41 loutilities kernel: [15183]    89 15183    22294      250      43        0             0 bounce
Aug  4 15:44:41 loutilities kernel: [15203]    48 15203   129291    26230     243      735             0 httpd
Aug  4 15:44:41 loutilities kernel: [15208]    48 15208   129086    25882     242      746             0 httpd
Aug  4 15:44:41 loutilities kernel: [15209]    48 15209   130739    27559     246      734             0 httpd
Aug  4 15:44:41 loutilities kernel: [15213]    48 15213   131244    28227     247      733             0 httpd
Aug  4 15:44:41 loutilities kernel: [15214]    48 15214   129285    26098     243      737             0 httpd
Aug  4 15:44:41 loutilities kernel: [15219]    48 15219   146235    24169     250      694             0 httpd
Aug  4 15:44:41 loutilities kernel: [15220]    48 15220   137945    15850     231      715             0 httpd
Aug  4 15:44:41 loutilities kernel: [15221]    48 15221   104921     1282     192     1180             0 httpd
Aug  4 15:44:41 loutilities kernel: [15222]    48 15222   133818    11754     226      718             0 httpd
Aug  4 15:44:41 loutilities kernel: [15223]    48 15223   146235    24130     250      695             0 httpd
Aug  4 15:44:41 loutilities kernel: [15224]    48 15224   133818    11753     226      719             0 httpd
Aug  4 15:44:41 loutilities kernel: [15226]    48 15226   112646     9765     210      746             0 httpd
Aug  4 15:44:41 loutilities kernel: [15229]    48 15229   104884     1230     191     1214             0 httpd
Aug  4 15:44:41 loutilities kernel: [15230]    48 15230   104921     1279     192     1179             0 httpd
Aug  4 15:44:41 loutilities kernel: [15231]    48 15231   104921     1273     192     1179             0 httpd
Aug  4 15:44:41 loutilities kernel: [15232]    48 15232   104921     1273     192     1179             0 httpd
Aug  4 15:44:41 loutilities kernel: [15233]    48 15233   104921     1275     192     1179             0 httpd
Aug  4 15:44:41 loutilities kernel: [15236]    48 15236   104921     1278     192     1183             0 httpd
Aug  4 15:44:41 loutilities kernel: [15237]    48 15237   104921     1279     192     1183             0 httpd
Aug  4 15:44:41 loutilities kernel: [15238]    48 15238   104921     1275     192     1184             0 httpd
Aug  4 15:44:41 loutilities kernel: [15240]    48 15240   112391     9558     209      751             0 httpd
Aug  4 15:44:41 loutilities kernel: [15241]    48 15241   104921     1280     192     1181             0 httpd
Aug  4 15:44:41 loutilities kernel: [15256]    48 15256   104921     1269     192     1182             0 httpd
Aug  4 15:44:41 loutilities kernel: [15258]    48 15258   104921     1271     192     1180             0 httpd
Aug  4 15:44:41 loutilities kernel: [15259]    48 15259   126804    23852     238      733             0 httpd
Aug  4 15:44:41 loutilities kernel: [15260]    48 15260   126940    23839     238      732             0 httpd
Aug  4 15:44:41 loutilities kernel: [15261]    48 15261   126813    23835     237      733             0 httpd
Aug  4 15:44:41 loutilities kernel: [15262]    48 15262   104921     1271     192     1183             0 httpd
Aug  4 15:44:41 loutilities kernel: [15263]    48 15263   104921     1279     192     1181             0 httpd
Aug  4 15:44:41 loutilities kernel: [15264]    48 15264   105533     1328     193     1180             0 httpd
Aug  4 15:44:41 loutilities kernel: [15265]    48 15265   105013     1388     192     1180             0 httpd
Aug  4 15:44:41 loutilities kernel: [15266]    48 15266   104921     1271     192     1183             0 httpd
Aug  4 15:44:41 loutilities kernel: [15268]    48 15268   104921     1270     192     1183             0 httpd
Aug  4 15:44:41 loutilities kernel: [15269]    48 15269   104921     1287     192     1183             0 httpd
Aug  4 15:44:41 loutilities kernel: [15270]    48 15270   104991     1331     191     1209             0 httpd
Aug  4 15:44:41 loutilities kernel: [15271]    48 15271   104917     1275     192     1182             0 httpd
Aug  4 15:44:41 loutilities kernel: [15272]    48 15272   104956     1298     192     1178             0 httpd
Aug  4 15:44:41 loutilities kernel: [15273]    48 15273   104884     1233     191     1209             0 httpd
Aug  4 15:44:41 loutilities kernel: [15275]    48 15275   104884     1228     191     1211             0 httpd
Aug  4 15:44:41 loutilities kernel: [15276]    48 15276   104921     1287     192     1183             0 httpd
Aug  4 15:44:41 loutilities kernel: [15278]    48 15278   114267    11444     213      747             0 httpd
Aug  4 15:44:41 loutilities kernel: [15280]    48 15280   104921     1289     192     1183             0 httpd
Aug  4 15:44:41 loutilities kernel: [15281]    48 15281   104921     1278     192     1176             0 httpd
Aug  4 15:44:41 loutilities kernel: [15282]    48 15282   104921     1287     192     1183             0 httpd
Aug  4 15:44:41 loutilities kernel: [15283]    48 15283   104921     1278     192     1176             0 httpd
Aug  4 15:44:41 loutilities kernel: [15284]    48 15284   104921     1278     192     1176             0 httpd
Aug  4 15:44:41 loutilities kernel: [15313]   996 15313    28280       44      11        0             0 sh
Aug  4 15:44:41 loutilities kernel: [15314]   996 15314    26992       58       9        0             0 df
Aug  4 15:44:41 loutilities kernel: Out of memory: Kill process 26718 (mysqld) score 43 or sacrifice child
Aug  4 15:44:41 loutilities kernel: Killed process 26718 (mysqld) total-vm:1652016kB, anon-rss:30216kB, file-rss:0kB
Aug  4 15:44:41 loutilities mysqld_safe: /usr/bin/mysqld_safe: line 183: 26718 Killed                  nohup /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock < /dev/null >> /var/log/mariadb/mariadb.log 2>&1 >> /var/log/mariadb/mariadb.log 2>&1
Aug  4 15:44:42 loutilities mysqld_safe: 170804 15:44:42 mysqld_safe Number of processes running now: 0
Aug  4 15:44:42 loutilities mysqld_safe: 170804 15:44:42 mysqld_safe mysqld restarted
Aug  4 15:44:43 loutilities mysqld_safe: 170804 15:44:43 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended

[louking]

2017-08-11 14:44 sudo apachectl restart
2017-08-11 14:43 sudo vim /etc/httpd/sites-available/scoretility.com.conf
                  WSGIDaemonProcess rrwebapp user=scoretility group=scoretility threads=2 [reduced threads from 5]
2017-08-11 14:42 ps aux: rabbitmq  3435  6.8  3.3 2832872
2017-08-11 14:35 sudo shutdown -r now
2017-08-11 14:30 sudo vim /etc/rabbitmq/rabbitmq.config - [{rabbit, [{vm_memory_high_watermark, 0.1}]}]. (yes the period is needed)
2017-08-11 14:19 ps aux: rabbitmq   802 16.6  3.2 2834256
2017-08-11 14:17 sudo shutdown -r now
2017-08-11 14:14 ps aux: rabbitmq   810  1.5  3.5 2858368
2017-08-11 14:13 sudo supervisorctl restart all
2017-08-11 14:12 disable /etc/supervisor
                    sandbox.celeryd-longtask.conf-disabled
                    beta.celeryd.conf-disabled
                    sandbox.celeryd.conf-disabled
2017-08-11 14:12 ps aux: rabbitmq   810  1.5  4.0 2856320

before

**top - 06:04:36 up 1 day, 10:54,  1 user,  load average: 0.52, 0.55, 0.49
Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.3 us,  1.4 sy,  0.0 ni, 72.1 id,  0.3 wa,  0.0 hi,  0.0 si,  6.0 st
KiB Mem :  2048988 total,    66960 free,  1340908 used,   641120 buff/cache
KiB Swap:  1048572 total,   349996 free,   698576 used.   564884 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  817 rabbitmq  20   0 2868256  35968   1408 S   1.7  1.8  34:41.85 beam.smp
 1555 mysql     20   0 1649972  24428   1528 S   1.3  1.2  85:14.89 mysqld
 1290 root      20   0 1442132   6540   1364 S   0.0  0.3   8:17.86 fail2ban-server
18187 scoreti+  20   0  757016  16712    508 S   0.0  0.8   0:05.00 httpd
23467 apache    20   0  613620 125340   2568 S   0.0  6.1   0:28.85 httpd
18248 apache    20   0  612224 122200   5028 S   0.0  6.0   4:53.16 httpd
20714 apache    20   0  608904 119256   2868 S   0.0  5.8   2:37.12 httpd
22585 apache    20   0  602940 110452   2996 S   0.0  5.4   1:50.26 httpd
18182 scoreti+  20   0  599824   1580    144 S   0.0  0.1   0:00.48 httpd
18183 scoreti+  20   0  599824   1740    260 S   0.0  0.1   0:00.50 httpd
18184 scoreti+  20   0  599824   1584    144 S   0.0  0.1   0:00.47 httpd
18185 scoreti+  20   0  599824   1580    144 S   0.0  0.1   0:00.47 httpd
23890 apache    20   0  594088 122196   4896 S   0.0  6.0   0:12.35 httpd
18188 steeple+  20   0  583432   1604    144 S   0.0  0.1   0:00.44 httpd
18186 steeple+  20   0  567040   1632    148 S   0.0  0.1   0:00.47 httpd
  810 root      20   0  553264    420    156 S   0.0  0.0   0:16.61 tuned
  464 polkitd   20   0  528276    252      0 S   0.0  0.0   0:00.39 polkitd
23474 apache    20   0  528168 116688   2832 S   0.0  5.7   0:34.25 httpd
23682 apache    20   0  526912 118140   5172 S  46.8  5.8   0:45.43 httpd
23889 apache    20   0  526768 120520   5248 S   0.0  5.9   0:22.89 httpd
23925 apache    20   0  526348 120156   5020 S   0.0  5.9   0:16.35 httpd
23683 apache    20   0  525884 117192   5140 S   0.0  5.7   0:43.48 httpd
  815 root      20   0  417380   1944    320 S   0.0  0.1   0:14.12 httpd
 2713 scoreti+  20   0  384232   1500      0 S   0.0  0.1   0:20.44 celery
  502 root      20   0  331520   1352    372 S   0.0  0.1   0:04.48 firewalld
 2099 scoreti+  20   0  291124   8924   1184 S   0.3  0.4   3:15.58 celery
 2100 scoreti+  20   0  291120   7032   1244 S   0.0  0.3   3:22.72 celery
 2095 scoreti+  20   0  290948   9268   1232 S   0.3  0.5   3:10.42 celery
 2101 scoreti+  20   0  290916   8976   1272 S   0.0  0.4   3:13.17 celery
18180 apache    20   0  290036   1236      4 S   0.0  0.1   0:00.18 httpd
 2091 scoreti+  20   0  289856   8432   1144 S   0.3  0.4   2:32.86 celery
 2715 scoreti+  20   0  289824   1380      0 S   0.0  0.1   0:00.00 celery
 2716 scoreti+  20   0  289824   1488      0 S   0.0  0.1   0:00.00 celery
 2712 scoreti+  20   0  289812   1580      0 S   0.0  0.1   0:00.00 celery
 2714 scoreti+  20   0  288840   1204      0 S   0.0  0.1   0:00.00 celery
  816 root      20   0  285312    432    264 S   0.0  0.0   0:01.85 rsyslogd
 1604 root      20   0  221508   1812    440 S   0.0  0.1   0:45.99 supervisord**

after

top - 14:45:46 up 9 min,  1 user,  load average: 0.20, 0.29, 0.15
Tasks: 114 total,   2 running, 112 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.2 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem :  2048988 total,   518676 free,   759628 used,   770684 buff/cache
KiB Swap:  1048572 total,   785524 free,   263048 used.  1144672 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3435 rabbitmq  20   0 2834920  68476   1552 S   0.3  3.3   0:06.74 beam.smp
 1207 root      20   0 1442132  14924   2004 S   0.7  0.7   0:02.85 fail2ban-server
 1538 mysql     20   0 1233760  23236   3380 S   0.3  1.1   0:19.85 mysqld
 3897 scoreti+  20   0  599824  12096   2368 S   0.0  0.6   0:00.01 httpd
 3898 scoreti+  20   0  599824  12096   2368 S   0.0  0.6   0:00.02 httpd
 3899 scoreti+  20   0  599824  12096   2368 S   0.0  0.6   0:00.02 httpd
 3900 scoreti+  20   0  599700  14148   2368 S   0.0  0.7   0:00.01 httpd
 3904 apache    20   0  597776 122064   9832 S   0.0  6.0   0:01.80 httpd
 3905 apache    20   0  585840 119044   7248 S   0.0  5.8   0:02.59 httpd
 3903 steeple+  20   0  583308  12100   2368 S   0.0  0.6   0:00.02 httpd
 3902 scoreti+  20   0  575112  12100   2368 S   0.3  0.6   0:00.02 httpd
 3901 steeple+  20   0  566916  12100   2368 S   0.0  0.6   0:00.02 httpd
  808 root      20   0  553264   1168    908 S   0.0  0.1   0:00.26 tuned
  429 polkitd   20   0  528276   1576   1116 S   0.0  0.1   0:00.06 polkitd
 3906 apache    20   0  522516 120720   6896 S   0.0  5.9   0:03.16 httpd
 3908 apache    20   0  519988 117636   6840 S   0.0  5.7   0:01.70 httpd
 3959 apache    20   0  515868 113464   6784 S   0.0  5.5   0:01.33 httpd
 3950 apache    20   0  421620  17720   5744 S   0.0  0.9   0:00.12 httpd
 3907 apache    20   0  419544  12808   3040 S   0.0  0.6   0:00.01 httpd
 3952 apache    20   0  419544  12808   3036 S   0.0  0.6   0:00.01 httpd
 3957 apache    20   0  419544  12808   3040 S   0.0  0.6   0:00.01 httpd
 3894 root      20   0  417368  18936  11396 S   0.0  0.9   0:00.20 httpd
  471 root      20   0  331520   2256   1304 S   0.0  0.1   0:01.01 firewalld
 2009 scoreti+  20   0  292504   9312   1568 S   0.3  0.5   0:02.08 celery
 2008 scoreti+  20   0  292436  11348   1524 S   0.7  0.6   0:02.03 celery
 2813 scoreti+  20   0  289816   2456    788 S   0.0  0.1   0:00.00 celery
 2814 scoreti+  20   0  289816   2500    788 S   0.0  0.1   0:00.00 celery
  806 root      20   0  285312   1712   1512 S   0.0  0.1   0:00.11 rsyslogd
 3895 apache    20   0  276556   5084    920 S   0.0  0.2   0:00.00 httpd
 1559 root      20   0  221472   1860    752 S   0.0  0.1   0:00.15 supervisord

[louking]

created new digitalocean volume (10GB). need to convert this to swap partition

sudo mkfs.ext4 -F /dev/disk/by-id/scsi-0DO_Volume_loutility-server-swap
sudo mkdir -p /mnt/loutility-server-swap
sudo mount -o discard,defaults /dev/disk/by-id/scsi-0DO_Volume_loutility-server-swap /mnt/loutility-server-swap
echo /dev/disk/by-id/scsi-0DO_Volume_loutility-server-swap /mnt/loutility-server-swap ext4 defaults,nofail,discard 0 0 | sudo tee -a /etc/fstab

[louking]

Unmount new volume, create new swap partition, remove old swap file

sudo umount /mnt/loutility-server-swap/
### see https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-swap-adding.html
sudo mkswap /dev/disk/by-id/scsi-0DO_Volume_loutility-server-swap
sudo vim /etc/fstab
   /dev/disk/by-id/scsi-0DO_Volume_loutility-server-swap swap swap defaults 0 0
sudo swapon -va
sudo swapoff -v /swapfile
sudo vim /etc/fstab [remove /swapfile none swap sw 0 0]
sudo rm /swapfile

[louking]

messages - out of memory, causing mariadb to be shut down

memory-crash-2017-08-13-20-09.txt memory-crash-2017-08-13-20-09.xlsx

is the following causing this?

184.168.46.179 - - [13/Aug/2017:20:08:07 -0400] "POST /wp-cron.php?doing_wp_cron=1502669286.7592020034790039062500 HTTP/1.1" 301 294 "http://steeplechasers.org/wp-cron.php?doing_wp_cron=1502669286.7592020034790039062500" "WordPress/4.8; https://steeplechasers.org"
46.229.168.78 - - [13/Aug/2017:20:08:13 -0400] "GET /about/inspirational-messages/ HTTP/1.1" 301 264 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"

(first memory crash was just after that GET) see memory-crash-2017-08-13-20-08.txt

mariadb crash

170813 16:22:38 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170813 16:23:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
170813 16:23:27 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 1549 ...
170813 16:23:27 InnoDB: The InnoDB memory heap is disabled
170813 16:23:27 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170813 16:23:27 InnoDB: Compressed tables use zlib 1.2.7
170813 16:23:27 InnoDB: Using Linux native AIO
170813 16:23:27 InnoDB: Initializing buffer pool, size = 128.0M
170813 16:23:27 InnoDB: Completed initialization of buffer pool
170813 16:23:27 InnoDB: highest supported file format is Barracuda.
170813 16:23:28  InnoDB: Waiting for the background threads to start
170813 16:23:29 Percona XtraDB (http://www.percona.com) 5.5.49-MariaDB-38.0 started; log sequence number 10653283285
170813 16:23:29 [Note] Plugin 'FEEDBACK' is disabled.
170813 16:23:29 [Note] Server socket created on IP: '0.0.0.0'.
170813 16:23:29 [Note] Event Scheduler: Loaded 0 events
170813 16:23:29 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
170813 20:09:18 mysqld_safe Number of processes running now: 0
170813 20:09:18 mysqld_safe mysqld restarted
170813 20:09:39 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 15218 ...
170813 20:09:43 InnoDB: The InnoDB memory heap is disabled
170813 20:09:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170813 20:09:43 InnoDB: Compressed tables use zlib 1.2.7
170813 20:09:43 InnoDB: Using Linux native AIO
170813 20:09:43 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
170813 20:09:43 InnoDB: Completed initialization of buffer pool
170813 20:09:43 InnoDB: Fatal error: cannot allocate memory for the buffer pool
170813 20:09:43 [ERROR] Plugin 'InnoDB' init function returned error.
170813 20:09:43 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
170813 20:09:43 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
170813 20:09:53 [Note] Plugin 'FEEDBACK' is disabled.
170813 20:09:53 [ERROR] Unknown/unsupported storage engine: InnoDB
170813 20:09:53 [ERROR] Aborting

170813 20:09:53 [Note] /usr/libexec/mysqld: Shutdown complete

170813 20:09:55 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended

[louking]

I've added some instrumentation (netstat and mysql show process list) and also apache restart to the cron job which runs every minute to check memory.

The netstat command yielded some information. Looks like a lot of connections get stuck in CLOSE_WAIT state.

The show processlist didn't seem to have any good information at the time I restarted apache. The snapshots I got earlier when the memory started depleting were similar.

The offending host this time was out of WA. Will watch if there is any consistency for the hosts which get me into this state. I did see some logs from this host in the access_log this time, and it is a site vulnerability detector gravityscan.com. Interesting, as I was watching the access log the other day when gravityscan was running and my server did not get into this state, and the scan my sites every night. This is a new service, and I wonder if they have bugs?

I want to know if there is any correlation with when the server started crashing and when I started using gravityscan. First mention of gravityscan in my logs is when I tried to install their agent in August, but I know the first time this event occurred was in June the night of our summer solstice race. I don't have in the log when I configured using their free service, and their full scan history is only available if we pay for the subscription.

---------- Forwarded message ---------- From: (Cron Daemon) root@scoretility.com Date: Fri, Oct 6, 2017 at 8:38 PM Subject: Cron root@loutilities /root/bin/checkswap To: webmaster@steeplechasers.org

Free swap 956324 below threshold 2000000

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:mysql           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:epmd            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:15672           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:smtp          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:33963         127.0.0.1:mysql         ESTABLISHED
tcp        0      0 198.199.82.162:epmd     198.199.82.162:50487    TIME_WAIT
tcp        0      0 127.0.0.1:mysql         127.0.0.1:33908         ESTABLISHED
tcp        0      0 127.0.0.1:33933         127.0.0.1:mysql         ESTABLISHED
tcp        0      0 127.0.0.1:epmd          127.0.0.1:56225         ESTABLISHED
tcp        0      0 127.0.0.1:56225         127.0.0.1:epmd          ESTABLISHED
tcp        0      0 127.0.0.1:33914         127.0.0.1:mysql         ESTABLISHED
tcp        0      0 127.0.0.1:mysql         127.0.0.1:33914         ESTABLISHED
tcp        0      0 127.0.0.1:37458         127.0.0.1:amqp          ESTABLISHED
tcp        0      0 127.0.0.1:37456         127.0.0.1:amqp          ESTABLISHED
tcp        0      0 127.0.0.1:33912         127.0.0.1:mysql         ESTABLISHED
tcp        0      0 127.0.0.1:33908         127.0.0.1:mysql         ESTABLISHED
tcp        0      0 127.0.0.1:mysql         127.0.0.1:33963         ESTABLISHED
tcp        0      0 127.0.0.1:mysql         127.0.0.1:33912         ESTABLISHED
tcp        0      0 127.0.0.1:mysql         127.0.0.1:33933         ESTABLISHED
tcp        0      0 127.0.0.1:37457         127.0.0.1:amqp          ESTABLISHED
tcp        0      0 127.0.0.1:37459         127.0.0.1:amqp          ESTABLISHED
tcp6       0      0 :::amqp                 :::*                    LISTEN
tcp6       0      0 :::http                 :::*                    LISTEN
tcp6       0      0 :::epmd                 :::*                    LISTEN
tcp6       0      0 :::ssh                  :::*                    LISTEN
tcp6       0      0 ::1:smtp                :::*                    LISTEN
tcp6       0      0 :::https                :::*                    LISTEN
tcp6       0      0 198.199.82.162:https    68.64.48.22:47242       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:51650       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:51634       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40064       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40644       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41628       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:48078       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    5.9.32.222:34050        ESTABLISHED
tcp6       0      0 127.0.0.1:amqp          127.0.0.1:37459         ESTABLISHED
tcp6       0      0 198.199.82.162:https    68.64.48.22:49024       ESTABLISHED
tcp6       1      0 198.199.82.162:https    68.64.48.22:42084       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:56974       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47218       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47124       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48284       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:59614       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:49020       ESTABLISHED
tcp6       1      0 198.199.82.162:https    68.64.48.22:49676       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:42622       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47216       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:46324       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47066       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:49032       ESTABLISHED
tcp6       1      0 198.199.82.162:https    68.64.48.22:33196       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40200       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:49004       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:34956       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45246       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40794       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43312       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40258       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:32948       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40002       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40954       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:37880       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46774       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:57540       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:59664       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38416       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:56804       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:42010       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40228       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:44040       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47016       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39712       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46868       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40170       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48986       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41642       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46302       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47144       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41496       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38506       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47106       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41020       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38702       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:37888       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:35476       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46874       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39430       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:56892       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38368       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39648       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:47286       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40062       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.:crestron-cips CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47056       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47122       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:36974       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38186       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:54440       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:46384       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:59668       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48660       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:54024       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45082       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39094       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:48146       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46970       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47234       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47008       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46912       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:52648       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41576       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:47260       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46364       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47262       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:37656       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39358       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:49114       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38446       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47068       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46920       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45732       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:35726       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:36212       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:44876       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:58904       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47240       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:46578       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:52528       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46748       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48306       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:35102       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45668       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47080       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:36282       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:33628       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:46482       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48298       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:59788       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:50668       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46852       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45230       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46972       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38852       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43416       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40970       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:59530       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46850       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:37484       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47166       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46950       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:35408       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:33542       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:34880       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38306       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48314       ESTABLISHED
tcp6       0      0 198.199.82.162:https    68.64.48.22:48672       ESTABLISHED
tcp6       1      0 198.199.82.162:https    68.64.48.22:42590       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41592       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47038       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41474       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:48496       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:56906       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38218       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47248       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48962       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:56316       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:49744       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:48040       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:56388       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:54430       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43734       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47592       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47188       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48336       ESTABLISHED
tcp6       1      0 198.199.82.162:https    68.64.48.22:42240       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40458       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:46132       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41484       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41940       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43576       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:32970       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:54458       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:42966       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:58976       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:35512       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48644       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:57750       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:59220       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46884       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:34798       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:33010       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:42202       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41446       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43762       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:47678       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:36694       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:33654       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43692       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:36456       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47226       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:36246       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39984       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:44644       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38268       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47608       ESTABLISHED
tcp6       1      0 198.199.82.162:https    68.64.48.22:38188       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:56966       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48994       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:35410       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46794       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:46162       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:37376       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39016       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46828       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43272       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39028       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40796       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46930       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47194       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:36638       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46900       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:52478       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43696       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:54426       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:42534       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:36436       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:37936       CLOSE_WAIT
tcp6       0      0 127.0.0.1:amqp          127.0.0.1:37457         ESTABLISHED
tcp6       1      0 198.199.82.162:https    68.64.48.22:48122       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:54070       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:42492       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:41140       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:59542       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40132       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45250       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40034       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:44174       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:37354       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:44624       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:60228       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43634       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43600       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47036       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45184       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:47710       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:44832       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:47276       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:42646       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:46772       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:55202       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46904       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45154       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:37710       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:34870       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:34890       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:34082       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47030       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47018       TIME_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47162       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:42082       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:48264       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43426       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:43610       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47092       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:35524       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45296       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:46814       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:57004       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:53296       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:37418       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:49772       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:49588       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:49132       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:52602       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:58892       CLOSE_WAIT
tcp6       0      0 127.0.0.1:amqp          127.0.0.1:37456         ESTABLISHED
tcp6       1      0 198.199.82.162:https    68.64.48.22:56976       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:46634       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47140       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39670       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:38366       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:46516       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:45170       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:39186       CLOSE_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:35428       CLOSE_WAIT
tcp6       0      0 198.199.82.162:https    68.64.48.22:47094       TIME_WAIT
tcp6       1      0 198.199.82.162:https    68.64.48.22:40928       CLOSE_WAIT
tcp6       0      0 127.0.0.1:amqp          127.0.0.1:37458         ESTABLISHED
tcp6       0      0 198.199.82.162:https    68.64.48.22:47272       TIME_WAIT
+-------+----------------+-----------------+--------------------+---------+------+-------+------------------+----------+
| Id    | User           | Host            | db                 | Command | Time | State | Info             | Progress |
+-------+----------------+-----------------+--------------------+---------+------+-------+------------------+----------+
| 42680 | scoretility    | localhost:33908 | scoretilitysandbox | Sleep   | 1653 |       |                  | 0.000    |
| 42694 | scoretility    | localhost:33912 | scoretilitysandbox | Sleep   | 1572 |       |                  | 0.000    |
| 42701 | scoretility    | localhost:33914 | scoretility        | Sleep   | 1558 |       |                  | 0.000    |
| 42800 | scoretility    | localhost:33933 | scoretility        | Sleep   | 945  |       |                  | 0.000    |
| 42851 | scoretility    | localhost:33963 | scoretilitysandbox | Sleep   | 671  |       |                  | 0.000    |
| 42941 | steeplechasers | localhost       | steeplechasers.org | Sleep   | 0    |       |                  | 0.000    |
| 42943 | root           | localhost       |                    | Query   | 0    |       | show processlist | 0.000    |
+-------+----------------+-----------------+--------------------+---------+------+-------+------------------+----------+

22:38:02: Restarting apache 22:39:34: Apache restart complete