vasukolli commented 7 years ago

Required information

Distribution: ubuntu
Distribution version: 16.04
The output of "lxc info" or if that fails: config: {} apiextensions:
- id_map apistatus: stable apiversion: "1.0" auth: trusted public: false environment: addresses: [] architectures:
- x86_64
- i686 driver: lxc driverversion: 2.0.7 kernel: Linux kernelarchitecture: x86_64 kernelversion: 4.4.0-47-generic server: lxd serverpid: 1337 serverversion: 2.0.9 storage: dir storageversion: ""

Issue description

Stotage filled up due to a log file in a container, so I stopped it and deleted the log file i..e(this log file folder is added as a device from host machine) and then started the container. Host machine (ubuntu 16.04) running 13 containers. after issuing top command I found that lxd is consuming more cpu power (134%)

Is this usage rate is expected ?

top -c -p 1337

top - 12:30:20 up 102 days, 2:45, 3 users, load average: 1.35, 1.41, 1.63 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie %Cpu(s): 10.8 us, 0.4 sy, 0.0 ni, 77.5 id, 0.0 wa, 0.0 hi, 0.0 si, 11.2 st KiB Mem : 24689584 total, 984240 free, 17741428 used, 5963916 buff/cache KiB Swap: 524284 total, 37248 free, 487036 used. 5783044 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1337 root 20 0 4458604 63972 11108 S 134.9 0.3 75:25.07 /usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log

Steps to reproduce

Information to attach

[ ] any relevant kernel output (dmesg)
[x] container log (lxc info NAME --show-log) lxc info gimboard-somename--show-log Name: gimboard-somename Remote: unix:/var/lib/lxd/unix.socket Architecture: x86_64 Created: 2016/06/18 10:47 UTC Status: Running Type: persistent Profiles: java-tomcat Pid: 19393 Ips: eth0: inet 10.186.93.94 veth6IKC1C eth0: inet6 fe80::216:3eff:fe2e:75a7 veth6IKC1C lo: inet 127.0.0.1 lo: inet6 ::1 Resources: Processes: 180 Memory usage: Memory (current): 1.60GB Memory (peak): 1.61GB Network usage: eth0: Bytes received: 350.64kB Bytes sent: 9.28MB Packets received: 2690 Packets sent: 2476 lo: Bytes received: 28.47MB Bytes sent: 28.47MB Packets received: 60544 Packets sent: 60544

Log:

        lxc 20160227062319.656 WARN     lxc_start - start.c:signal_handler:322 - Invalid pid for SIGCHLD. Received pid 19387, expected pid 19393.

[x] main daemon log (/var/log/lxd.log) vl=info msg="Updating images" t=2017-02-27T08:48:51+0530 alias=debian/jessie/amd64 lvl=info msg="Downloading image" server=https://images.linuxcontainers.org t=2017-02-27T08:48:58+0530 alias=debian/jessie/amd64 lvl=info msg="Image downloaded" server=https://images.linuxcontainers.org t=2017-02-27T08:49:16+0530 alias=centos/7/amd64 lvl=info msg="Downloading image" server=https://images.linuxcontainers.org t=2017-02-27T08:49:30+0530 alias=centos/7/amd64 lvl=info msg="Image downloaded" server=https://images.linuxcontainers.org t=2017-02-27T08:49:46+0530 lvl=info msg="Done updating images" t=2017-02-27T08:49:46+0530 action=shutdown creation date=2016-06-18T16:17:42+0530 ephemeral=false lvl=info msg="Shutting down container" name=gimboard-somename t=2017-02-27T11:38:34+0530 timeout=-1s action=stop created=2016-06-18T16:17:42+0530 ephemeral=false lvl=info msg="Container initiated stop" name=gimboard-somename stateful=false t=2017-02-27T11:39:40+0530 action=shutdown creation date=2016-06-18T16:17:42+0530 ephemeral=false lvl=info msg="Shut down container" name=gimboard-somename t=2017-02-27T11:39:40+0530 timeout=-1s action=R err="exit status 254" lvl=eror msg="Running apparmor" output="apparmorparser: Unable to remove \"lxd-gimboard-somename</var/lib/lxd>\". Profile doesn't exist\n" t=2017-02-27T11:39:4$ container=gimboard-somename err="exit status 254" lvl=eror msg="Failed to destroy apparmor namespace" t=2017-02-27T11:39:40+0530 action=start creation date=2016-06-18T16:17:42+0530 ephemeral=false lvl=info msg="Starting container" name=gimboard-somename stateful=false t=2017-02-27T11:53:19+0530 action=start creation date=2016-06-18T16:17:42+0530 ephemeral=false lvl=info msg="Started container" name=gimboard-somename stateful=false t=2017-02-27T11:53:19+0530
[ ] output of the client with --debug
[ ] output of the daemon with --debug

stgraber commented 7 years ago

That certainly looks excessive... Was that just a temporary spike or is LXD still burning all your CPU?

vasukolli commented 7 years ago

still burning. please find image lxd

stgraber commented 7 years ago

Can you run "lxc monitor --type=logging"? Let it run for a few seconds then cancel it and paste the output here. That may give us a clue as to what LXD's doing.

vasukolli commented 7 years ago

lxc monitor --type=logging output

metadata: context: {} level: dbug message: 'New events listener: 426637ae-4cad-48f3-b805-2a5ff0ee1c3e' timestamp: 2017-02-27T23:48:13.951846392+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.95185931+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951823514+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951873763+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.9518889+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951902462+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951915054+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951927969+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951940287+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951952475+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951969938+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.95202337+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951982871+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.951997044+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.952009687+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.952082436+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.952036098+05:30 type: logging

metadata: context: {} level: dbug message: Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event. timestamp: 2017-02-27T23:48:13.95205378+05:30 type: logging

stgraber commented 7 years ago

Looks like a very tight loop related to some background task executing in a container...

Can you paste "ps fauxww" from the LXD host?

vasukolli commented 7 years ago

ps fauxww

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S     2016   0:02 [kthreadd]
root         3  0.0  0.0      0     0 ?        S     2016   0:27  \_ [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kworker/0:0H]
root         7  0.0  0.0      0     0 ?        S     2016  42:31  \_ [rcu_sched]
root         8  0.0  0.0      0     0 ?        S     2016   0:00  \_ [rcu_bh]
root         9  0.0  0.0      0     0 ?        S     2016   0:24  \_ [migration/0]
root        10  0.0  0.0      0     0 ?        S     2016   0:38  \_ [watchdog/0]
root        11  0.0  0.0      0     0 ?        S     2016   0:38  \_ [watchdog/1]
root        12  0.0  0.0      0     0 ?        S     2016   0:27  \_ [migration/1]
root        13  0.0  0.0      0     0 ?        S     2016   0:33  \_ [ksoftirqd/1]
root        15  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kworker/1:0H]
root        16  0.0  0.0      0     0 ?        S     2016   0:38  \_ [watchdog/2]
root        17  0.0  0.0      0     0 ?        S     2016   0:29  \_ [migration/2]
root        18  0.0  0.0      0     0 ?        S     2016   0:28  \_ [ksoftirqd/2]
root        20  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kworker/2:0H]
root        21  0.0  0.0      0     0 ?        S     2016   0:40  \_ [watchdog/3]
root        22  0.0  0.0      0     0 ?        S     2016   0:26  \_ [migration/3]
root        23  0.0  0.0      0     0 ?        S     2016   1:01  \_ [ksoftirqd/3]
root        25  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kworker/3:0H]
root        26  0.0  0.0      0     0 ?        S     2016   0:37  \_ [watchdog/4]
root        27  0.0  0.0      0     0 ?        S     2016   0:24  \_ [migration/4]
root        28  0.0  0.0      0     0 ?        S     2016   0:23  \_ [ksoftirqd/4]
root        30  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kworker/4:0H]
root        31  0.0  0.0      0     0 ?        S     2016   0:38  \_ [watchdog/5]
root        32  0.0  0.0      0     0 ?        S     2016   0:21  \_ [migration/5]
root        33  0.0  0.0      0     0 ?        S     2016   0:28  \_ [ksoftirqd/5]
root        35  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kworker/5:0H]
root        36  0.0  0.0      0     0 ?        S     2016   0:38  \_ [watchdog/6]
root        37  0.0  0.0      0     0 ?        S     2016   0:20  \_ [migration/6]
root        38  0.0  0.0      0     0 ?        S     2016   0:21  \_ [ksoftirqd/6]
root        40  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kworker/6:0H]
root        41  0.0  0.0      0     0 ?        S     2016   0:37  \_ [watchdog/7]
root        42  0.0  0.0      0     0 ?        S     2016   0:24  \_ [migration/7]
root        43  0.0  0.0      0     0 ?        S     2016   0:23  \_ [ksoftirqd/7]
root        45  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kworker/7:0H]
root        46  0.0  0.0      0     0 ?        S     2016   0:00  \_ [kdevtmpfs]
root        47  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [netns]
root        48  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [perf]
root        49  0.0  0.0      0     0 ?        S     2016   0:23  \_ [khungtaskd]
root        50  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [writeback]
root        51  0.0  0.0      0     0 ?        SN    2016   0:00  \_ [ksmd]
root        52  0.0  0.0      0     0 ?        SN    2016   9:38  \_ [khugepaged]
root        53  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [crypto]
root        54  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kintegrityd]
root        55  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        56  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kblockd]
root        57  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [ata_sff]
root        58  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [md]
root        59  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [devfreq_wq]
root        63  0.0  0.0      0     0 ?        S     2016   8:03  \_ [kswapd0]
root        64  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [vmstat]
root        65  0.0  0.0      0     0 ?        S     2016   0:00  \_ [fsnotify_mark]
root        66  0.0  0.0      0     0 ?        S     2016   0:00  \_ [ecryptfs-kthrea]
root        82  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kthrotld]
root        83  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [acpi_thermal_pm]
root        84  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        85  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        86  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        87  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        88  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        89  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        90  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        91  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        92  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        93  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        94  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        95  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        96  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        97  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        98  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root        99  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       100  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       101  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       102  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       103  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       104  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       105  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       106  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       107  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       108  0.0  0.0      0     0 ?        S     2016   0:00  \_ [scsi_eh_0]
root       109  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [scsi_tmf_0]
root       110  0.0  0.0      0     0 ?        S     2016   0:00  \_ [scsi_eh_1]
root       111  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [scsi_tmf_1]
root       116  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [ipv6_addrconf]
root       130  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [deferwq]
root       131  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [charger_manager]
root       133  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       176  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       177  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       178  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       179  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       181  0.0  0.0      0     0 ?        S     2016   0:00  \_ [scsi_eh_2]
root       182  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [scsi_tmf_2]
root       183  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       184  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [kpsmoused]
root       186  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       187  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       189  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       190  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       192  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       195  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root       306  0.0  0.0      0     0 ?        S<    2016   0:52  \_ [kworker/1:1H]
root       481  0.0  0.0      0     0 ?        S     2016  12:04  \_ [jbd2/sda-8]
root       482  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [ext4-rsv-conver]
root       515  0.0  0.0      0     0 ?        S<    2016   0:18  \_ [kworker/3:1H]
root       535  0.0  0.0      0     0 ?        S     2016   0:00  \_ [kauditd]
root       620  0.0  0.0      0     0 ?        S<    2016   0:14  \_ [kworker/7:1H]
root       813  0.0  0.0      0     0 ?        S     2016   3:15  \_ [jbd2/sdc-8]
root       814  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [ext4-rsv-conver]
root       815  0.0  0.0      0     0 ?        S<    2016   0:18  \_ [kworker/0:1H]
root       925  0.0  0.0      0     0 ?        S<    2016   3:20  \_ [kworker/5:1H]
root      1141  0.0  0.0      0     0 ?        S<    2016   0:15  \_ [kworker/4:1H]
root      1757  0.0  0.0      0     0 ?        S<    2016   0:23  \_ [kworker/2:1H]
root      3536  0.0  0.0      0     0 ?        S<    2016   0:10  \_ [kworker/6:1H]
root      9891  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [xfsalloc]
root      9892  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [xfs_mru_cache]
root      9923  0.0  0.0      0     0 ?        S<    2016   0:00  \_ [bioset]
root     18743  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/7:0]
root     21133  0.0  0.0      0     0 ?        S    Feb27   0:01  \_ [kworker/1:1]
root     22435  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/2:2]
root     26037  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/0:1]
root     29553  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/3:0]
root     29572  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/6:2]
root       353  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/5:1]
root       354  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/3:1]
root       356  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/0:0]
root       365  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/2:3]
root       380  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/1:2]
root       397  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/6:0]
root      1276  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/4:0]
root      2368  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/u16:2]
root      3793  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/5:0]
root      3843  0.0  0.0      0     0 ?        S    Feb27   0:00  \_ [kworker/u16:1]
root      3964  0.0  0.0      0     0 ?        S    00:00   0:00  \_ [kworker/0:2]
root      3988  0.0  0.0      0     0 ?        S    00:00   0:00  \_ [kworker/1:3]
root      3996  0.0  0.0      0     0 ?        S    00:00   0:00  \_ [kworker/7:2]
root      4001  0.0  0.0      0     0 ?        S    00:00   0:00  \_ [kworker/4:1]
root      4107  0.0  0.0      0     0 ?        S    00:01   0:00  \_ [kworker/0:3]
root      4116  0.0  0.0      0     0 ?        S    00:01   0:00  \_ [kworker/6:1]
root      4166  0.0  0.0      0     0 ?        S    00:01   0:00  \_ [kworker/6:3]
root         1  0.0  0.0 185372  4256 ?        Ss    2016   3:06 /lib/systemd/systemd --system --deserialize 22
root       525  0.0  0.0  52056 21180 ?        Ss    2016   3:33 /lib/systemd/systemd-journald
root      1017  0.0  0.0  28548  1692 ?        Ss    2016   0:15 /lib/systemd/systemd-logind
root      1022  0.0  0.0  29008  1116 ?        Ss    2016   0:23 /usr/sbin/cron -f
root      1023  0.0  0.0 530244  1188 ?        Ssl   2016  55:17 /usr/bin/lxcfs /var/lib/lxcfs/
message+  1030  0.0  0.0  43000  1416 ?        Ss    2016   0:21 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root      1042  0.0  0.0 275872   788 ?        Ssl   2016   3:50 /usr/lib/accountsservice/accounts-daemon
syslog    1054  0.0  0.0 256396  1904 ?        Ssl   2016   1:51 /usr/sbin/rsyslogd -n
root      1119  0.0  0.0  16116  2176 ?        Ss    2016   0:00 /sbin/dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
root      1127  0.0  0.0  19540   264 ?        Ss    2016   4:42 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
root      1251  0.0  0.0  15940   140 tty1     Ss+   2016   0:00 /sbin/agetty --noclear tty1 linux
lxd       1258  0.0  0.0  52868  1364 ?        S     2016   1:21 dnsmasq -s lxd -S /lxd/ -u lxd --strict-order --bind-interfaces --pid-file=/run/lxd-bridge//dnsmasq.pid --dhcp-no-override --except-interface=lo --interface=lxdbr0 --dhcp-leasefile=/var/lib/lxd-bridge//dnsmasq.lxdbr0.leases --dhcp-authoritative --listen-address 10.186.93.1 --dhcp-range 10.186.93.2,10.186.93.254 --dhcp-lease-max=252
root      1317  0.0  0.0  73236   600 ?        Ss    2016   0:00 [lxc monitor] /var/lib/lxd/containers classy
165536    1349  0.0  0.0  37284  2072 ?        Ss    2016   0:36  \_ /sbin/init
165536    1415  0.0  0.0  35276  7936 ?        Ss    2016   0:26      \_ /lib/systemd/systemd-journald
165536    1418  0.0  0.0  41588   120 ?        Ss    2016   0:06      \_ /lib/systemd/systemd-udevd
165640    1492  0.0  0.0 182664   592 ?        Ssl   2016   0:05      \_ /usr/sbin/rsyslogd -n
165536    1500  0.0  0.0  29008   356 ?        Ss    2016   0:17      \_ /usr/sbin/cron -f
165643    1504  0.0  0.0  42892   316 ?        Ss    2016   0:00      \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536    1514  0.0  0.0  20100    96 ?        Ss    2016   0:07      \_ /lib/systemd/systemd-logind
165536    1581  0.0  0.0  16120   908 ?        Ss    2016   0:06      \_ /sbin/dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
165536    1610  0.0  0.0  15756   148 pts/0    Ss+   2016   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165642    1614  0.0  0.5 1345020 138840 ?      Ssl   2016 118:41      \_ /usr/sbin/mysqld
165536    1637  0.0  0.0 404344 15672 ?        Ss    2016   5:57      \_ /usr/sbin/apache2 -k start
165569   26785  0.0  0.0 404376  5792 ?        S    Feb27   0:00          \_ /usr/sbin/apache2 -k start
165569   26786  0.0  0.0 404376  5792 ?        S    Feb27   0:00          \_ /usr/sbin/apache2 -k start
165569   26787  0.0  0.0 404376  5792 ?        S    Feb27   0:00          \_ /usr/sbin/apache2 -k start
165569   26788  0.0  0.0 404376  5792 ?        S    Feb27   0:00          \_ /usr/sbin/apache2 -k start
165569   26789  0.0  0.0 404376  5792 ?        S    Feb27   0:00          \_ /usr/sbin/apache2 -k start
root      1367  0.0  0.1 320748 46200 ?        Sl    2016  54:14 /usr/bin/python3 /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b
root      2903  0.0  0.0  73236  1424 ?        Ss    2016   0:00 [lxc monitor] /var/lib/lxd/containers gimboard-kerala
165536    2923  0.0  0.0 190764  4860 ?        Ss    2016   1:29  \_ /usr/lib/systemd/systemd --system --deserialize 18
165536    2979  0.0  0.0  36828  8200 ?        Ss    2016   0:39      \_ /usr/lib/systemd/systemd-journald
165617    3003  0.0  0.0  26688  1816 ?        Ss    2016   0:13      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536    3011  0.0  0.0  26908  1768 ?        Ss    2016   0:29      \_ /usr/lib/systemd/systemd-logind
165536    3015  0.0  0.0   6464     8 pts/3    Ss+   2016   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165536    3055  0.2  3.7 6609784 938016 ?      Sl    2016 336:55      \_ /srv/softwares/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -server -XX:+UseParallelGC -classpath /srv/softwares/tomcat/bin/bootstrap.jar:/srv/softwares/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/srv/softwares/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
165536    3444  0.0  0.0 110536  1536 ?        Ss    2016   0:08      \_ /sbin/dhclient -H gimboard-kerala -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165536    9089  0.0  0.0  34844  2804 ?        Ss   Feb27   0:00      \_ /usr/lib/systemd/systemd-udevd
165536    9186  0.0  0.0  22640  2536 ?        Ss   Feb27   0:00      \_ /usr/sbin/crond -n
165563    9247  0.0  0.0   9516  2556 ?        Ss   Feb27   0:00      \_ /bin/sh /usr/bin/mysqld_safe --basedir=/usr
165563    9403  0.1  0.5 971556 134156 ?       Sl   Feb27   0:53      |   \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
165536    9469  0.0  0.0 271068  7964 ?        Ssl  Feb27   0:00      \_ /usr/sbin/rsyslogd -n
root      3551  0.0  0.0 148280  1452 ?        Ss    2016   0:00 [lxc monitor] /var/lib/lxd/containers gimboard-modern
165536    3574  0.0  0.0 190888  4748 ?        Ss    2016   1:28  \_ /usr/lib/systemd/systemd --system --deserialize 20
165536    3644  0.0  0.0  36828  8028 ?        Ss    2016   0:39      \_ /usr/lib/systemd/systemd-journald
165617    3698  0.0  0.0  26696  1304 ?        Ss    2016   0:13      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536    3735  0.0  0.0  27424  2072 ?        Ss    2016   0:29      \_ /usr/lib/systemd/systemd-logind
165536    3747  0.0  0.0   6464     4 pts/5    Ss+   2016   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165536    4145  0.0  0.0 110536  1052 ?        Ss    2016   0:08      \_ /sbin/dhclient -H gimboard-modern -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165536   20219  0.3  4.6 6631328 1156748 ?     Sl   Feb23  23:48      \_ /srv/softwares/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -server -XX:+UseParallelGC -classpath /srv/softwares/tomcat/bin/bootstrap.jar:/srv/softwares/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/srv/softwares/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
165536    6432  0.0  0.0  34844  2336 ?        Ss   Feb27   0:00      \_ /usr/lib/systemd/systemd-udevd
165536    6525  0.0  0.0  22644  2408 ?        Ss   Feb27   0:00      \_ /usr/sbin/crond -n
165563    6586  0.0  0.0   9516  2344 ?        Ss   Feb27   0:00      \_ /bin/sh /usr/bin/mysqld_safe --basedir=/usr
165563    6785  0.1  0.6 3670508 149184 ?      Sl   Feb27   1:27      |   \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
165536    6850  0.0  0.0 271068  6976 ?        Ssl  Feb27   0:00      \_ /usr/sbin/rsyslogd -n
root      5475  0.0  0.0  73236  1384 ?        Ss    2016   0:00 [lxc monitor] /var/lib/lxd/containers gimboard-msn
165536    5506  0.0  0.0 190860  5108 ?        Ss    2016   1:30  \_ /usr/lib/systemd/systemd --system --deserialize 19
165536    5553  0.0  0.0  36828  7976 ?        Ss    2016   0:39      \_ /usr/lib/systemd/systemd-journald
165617    5581  0.0  0.0  26688  1228 ?        Ss    2016   0:14      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536    5591  0.0  0.0  27420  2060 ?        Ss    2016   0:30      \_ /usr/lib/systemd/systemd-logind
165536    5599  0.0  0.0   6464     8 pts/1    Ss+   2016   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165536    5606  0.1  3.1 6565664 788464 ?      Sl    2016 290:02      \_ /srv/softwares/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -server -XX:+UseParallelGC -classpath /srv/softwares/tomcat/bin/bootstrap.jar:/srv/softwares/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/srv/softwares/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
165536    6057  0.0  0.0 110536  1184 ?        Ss    2016   0:08      \_ /sbin/dhclient -H gimboard-msn -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165536    8231  0.0  0.0  34844  2612 ?        Ss   Feb27   0:00      \_ /usr/lib/systemd/systemd-udevd
165536    8324  0.0  0.0  22640  2732 ?        Ss   Feb27   0:00      \_ /usr/sbin/crond -n
165563    8389  0.0  0.0   9516  2568 ?        Ss   Feb27   0:00      \_ /bin/sh /usr/bin/mysqld_safe --basedir=/usr
165563    8545  0.0  0.4 971556 114372 ?       Sl   Feb27   0:48      |   \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
165536    8609  0.0  0.0 271068  7948 ?        Ssl  Feb27   0:00      \_ /usr/sbin/rsyslogd -n
root      6145  0.0  0.0  73236  1368 ?        Ss    2016   0:00 [lxc monitor] /var/lib/lxd/containers gimboard-vijetha
165536    6167  0.0  0.0 190860  5084 ?        Ss    2016   1:30  \_ /usr/lib/systemd/systemd --system --deserialize 18
165536    6220  0.0  0.0  36828  8208 ?        Ss    2016   0:39      \_ /usr/lib/systemd/systemd-journald
165536    6244  0.0  0.0  27420  2228 ?        Ss    2016   0:30      \_ /usr/lib/systemd/systemd-logind
165617    6250  0.0  0.0  26684  1752 ?        Ss    2016   0:14      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536    6276  0.0  0.0   6464     8 pts/2    Ss+   2016   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165536    6300  0.1  3.2 6565816 794188 ?      Sl    2016 291:31      \_ /srv/softwares/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -server -XX:+UseParallelGC -classpath /srv/softwares/tomcat/bin/bootstrap.jar:/srv/softwares/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/srv/softwares/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
165536    6699  0.0  0.0 110536  1592 ?        Ss    2016   0:08      \_ /sbin/dhclient -H gimboard-vijetha -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165536   10648  0.0  0.0  34844  2780 ?        Ss   Feb27   0:00      \_ /usr/lib/systemd/systemd-udevd
165536   10750  0.0  0.0  22644  2616 ?        Ss   Feb27   0:00      \_ /usr/sbin/crond -n
165563   10853  0.0  0.0   9516  2608 ?        Ss   Feb27   0:00      \_ /bin/sh /usr/bin/mysqld_safe --basedir=/usr
165563   11010  0.0  0.4 971552 112672 ?       Sl   Feb27   0:48      |   \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
165536   11076  0.0  0.1 295644 36764 ?        Ssl  Feb27   0:02      \_ /usr/sbin/rsyslogd -n
root     19688  0.0  0.0  68908   524 ttyS0    Ss    2016   0:00 /bin/login --
greenbu+ 19715  0.0  0.0  22488  2632 ttyS0    S     2016   0:00  \_ -bash
root     18341  0.0  0.0  52284  3016 ttyS0    S    Feb27   0:00      \_ su root
root     18351  0.0  0.0  21196  3276 ttyS0    S+   Feb27   0:01          \_ bash
greenbu+ 19707  0.0  0.0  45356   960 ?        Ss    2016   0:00 /lib/systemd/systemd --user
greenbu+ 19711  0.0  0.0  61532  2232 ?        S     2016   0:00  \_ (sd-pam)
root     11477  0.0  0.0 148416  1476 ?        Ss    2016   0:00 [lxc monitor] /var/lib/lxd/containers texas-qa
165536   11493  0.0  0.0 190460  1692 ?        Ss    2016   0:17  \_ /sbin/init
165536   11555  0.0  0.0  36828  5732 ?        Ss    2016   0:11      \_ /usr/lib/systemd/systemd-journald
165536   11557  0.0  0.0  34844   456 ?        Ss    2016   0:00      \_ /usr/lib/systemd/systemd-udevd
165536   11590  0.0  0.0  22076   144 ?        Ss    2016   0:04      \_ /usr/lib/systemd/systemd-logind
165617   11591  0.0  0.0  24408   244 ?        Ss    2016   0:00      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536   11592  0.0  0.0 238300  1172 ?        Ssl   2016   0:02      \_ /usr/sbin/rsyslogd -n
165536   11600  0.0  0.0   6456    56 pts/13   Ss+   2016   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165536   11796  0.0  0.0 112832  5568 ?        Ss    2016   0:06      \_ /sbin/dhclient -H texas-qa -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165536   10463  0.2  7.6 11329280 1900436 ?    Sl    2016 207:29      \_ /opt/java/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
165563   17122  0.0  0.0   9516   284 ?        Ss    2016   0:00      \_ /bin/sh /usr/bin/mysqld_safe --basedir=/usr
165563   17426  0.1  1.7 3515144 443864 ?      Sl    2016 155:15          \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
root       491  0.0  0.0  73372  1384 ?        Ss    2016   0:00 [lxc monitor] /var/lib/lxd/containers texas-qa1
165536     519  0.0  0.0  42956  1800 ?        Ss    2016   0:17  \_ /sbin/init
165536     573  0.0  0.0  36828  5572 ?        Ss    2016   0:11      \_ /usr/lib/systemd/systemd-journald
165536     574  0.0  0.0  41400   600 ?        Ss    2016   0:00      \_ /usr/lib/systemd/systemd-udevd
165617     596  0.0  0.0  24416   260 ?        Ss    2016   0:00      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536     598  0.0  0.0  22076   200 ?        Ss    2016   0:04      \_ /usr/lib/systemd/systemd-logind
165536     602  0.0  0.0 238304  2208 ?        Ssl   2016   0:02      \_ /usr/sbin/rsyslogd -n
165536     604  0.0  0.0   6456   128 pts/7    Ss+   2016   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165563     660  0.0  0.0   9516   184 ?        Ss    2016   0:00      \_ /bin/sh /usr/bin/mysqld_safe --basedir=/usr
165563     909  0.0  0.2 840404 67780 ?        Sl    2016  84:41      |   \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
165536    1104  0.0  0.0 112832 12148 ?        Ss    2016   0:07      \_ /sbin/dhclient -H texas-qa1 -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165536    1626  0.1  0.5 10063544 136132 ?     Sl    2016 142:11      \_ /opt/java/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
root     14909  0.0  0.0 148428  1468 ?        Ss   Jan11   0:00 [lxc monitor] /var/lib/lxd/containers gimboard-kcp
165536   14933  0.0  0.0 190628  2476 ?        Ss   Jan11   0:17  \_ /sbin/init
165536   14987  0.0  0.0  45020 11164 ?        Ss   Jan11   0:11      \_ /usr/lib/systemd/systemd-journald
165536   14990  0.0  0.0  34844   744 ?        Ss   Jan11   0:00      \_ /usr/lib/systemd/systemd-udevd
165536   15015  0.0  0.0 246496  5688 ?        Ssl  Jan11   0:03      \_ /usr/sbin/rsyslogd -n
165617   15017  0.0  0.0  24396   432 ?        Ss   Jan11   0:06      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536   15028  0.0  0.0  24200  1196 ?        Ss   Jan11   0:06      \_ /usr/lib/systemd/systemd-logind
165536   15037  0.0  0.0   6456    12 pts/20   Ss+  Jan11   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165536   15439  0.0  0.0 112832  7780 ?        Ss   Jan11   0:04      \_ /sbin/dhclient -H gimboard-kcp -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165536   25946  0.0  0.0  22676  1092 ?        Ss   Jan16   0:06      \_ /usr/sbin/crond -n
165563   12303  0.0  0.0   9516   396 ?        Ss   Jan31   0:00      \_ /bin/sh /usr/bin/mysqld_safe --basedir=/usr
165563   12459  0.0  0.4 3029356 122792 ?      Sl   Jan31  39:07      |   \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
165536   12577  0.2  4.3 6597464 1073108 ?     Sl   Jan31  90:19      \_ /srv/softwares/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -server -XX:+UseParallelGC -classpath /srv/softwares/tomcat/bin/bootstrap.jar:/srv/softwares/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/srv/softwares/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
root     29744  0.0  0.0 148428  2008 ?        Ss   Jan11   0:00 [lxc monitor] /var/lib/lxd/containers greenbuds-invoice
165536   29764  0.0  0.0  37264  3976 ?        Ss   Jan11   0:17  \_ /lib/systemd/systemd --system --deserialize 19
165536   29840  0.0  0.0  35276  9036 ?        Ss   Jan11   0:09      \_ /lib/systemd/systemd-journald
165640   29948  0.0  0.0 182664   432 ?        Ssl  Jan11   0:02      \_ /usr/sbin/rsyslogd -n
165643   29951  0.0  0.0  42892  1900 ?        Ss   Jan11   0:00      \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536   29957  0.0  0.0  28980   500 ?        Ss   Jan11   0:07      \_ /usr/sbin/cron -f
165536   29958  0.0  0.0  20100   620 ?        Ss   Jan11   0:03      \_ /lib/systemd/systemd-logind
165536   30097  0.0  0.0  16124  1436 ?        Ss   Jan11   0:03      \_ /sbin/dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
165536   30130  0.0  0.0  15756    76 pts/10   Ss+  Jan11   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165536   19098  0.0  0.0  41588  2572 ?        Ss   Feb21   0:00      \_ /lib/systemd/systemd-udevd
165642   19414  0.0  1.0 1679244 246916 ?      Ssl  Feb21   7:46      \_ /usr/sbin/mysqld
165536   21820  0.0  0.0 451640  7388 ?        Ssl  Feb23   0:02      \_ Passenger watchdog
165536   21823  0.1  0.0 1929412 18952 ?       Sl   Feb23   9:36      |   \_ Passenger core
231070   21833  0.0  0.0 525736 10144 ?        Sl   Feb23   0:10      |   \_ Passenger ust-router
165536   21859  0.0  0.0 178460  6092 ?        Ss   Feb23   0:00      \_ nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
165569   21860  0.0  0.0 178460  7108 ?        S    Feb23   0:16      |   \_ nginx: worker process
165569   21861  0.0  0.0 178460  7204 ?        S    Feb23   0:38      |   \_ nginx: worker process
165569   21862  0.0  0.0 178460  7204 ?        S    Feb23   0:37      |   \_ nginx: worker process
165569   21863  0.0  0.0 178460  7108 ?        S    Feb23   0:22      |   \_ nginx: worker process
165569   21864  0.0  0.0 178460  7204 ?        S    Feb23   0:37      |   \_ nginx: worker process
165569   21865  0.0  0.0 178460  7108 ?        S    Feb23   0:36      |   \_ nginx: worker process
165569   21866  0.0  0.0 178460  7208 ?        S    Feb23   0:38      |   \_ nginx: worker process
165569   21867  0.0  0.0 178460  7208 ?        S    Feb23   0:37      |   \_ nginx: worker process
166536   21951  0.0  0.6 460132 150084 ?       Sl   Feb23   0:06      \_ Passenger RubyApp: /srv/appcontent/appdir/InvoiceStack/public (production)
uuidd    10097  0.0  0.0  28156   200 ?        Ss   Jan20   0:00 /usr/sbin/uuidd --socket-activation
systemd+ 19047  0.0  0.0 100324  2760 ?        Ssl  Jan23   0:03 /lib/systemd/systemd-timesyncd
root     19642  0.0  0.0  42072  2396 ?        Ss   Jan23   0:02 /lib/systemd/systemd-udevd
root     20010  0.0  0.0 148428  1644 ?        Ss   Jan23   0:00 [lxc monitor] /var/lib/lxd/containers improwise-dev
165536   20035  0.0  0.0  41600  3328 ?        Ss   Jan23   0:31  \_ /sbin/init
165536   20089  0.0  0.0  36828  6520 ?        Ss   Jan23   0:14      \_ /usr/lib/systemd/systemd-journald
165536   20090  0.0  0.0  41392   712 ?        Ss   Jan23   0:00      \_ /usr/lib/systemd/systemd-udevd
165536   20116  0.0  0.0  26408  1504 ?        Ss   Jan23   0:10      \_ /usr/lib/systemd/systemd-logind
165617   20118  0.0  0.0  26472   828 ?        Ss   Jan23   0:04      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536   20148  0.0  0.0 221892   524 ?        Ssl  Jan23   0:00      \_ /usr/sbin/rsyslogd -n
165536   20153  0.0  0.0   6464   120 pts/24   Ss+  Jan23   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165536   20154  0.0  0.0  22764  1200 ?        Ss   Jan23   0:06      \_ /usr/sbin/crond -n
165563   20185  0.0  1.3 1926892 327580 ?      Sl   Jan23  44:17      \_ /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
165536   20385  0.0  0.0 110536 13732 ?        Ss   Jan23   0:03      \_ /sbin/dhclient -H improwise-dev -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165536   17255  1.1 10.9 12297560 2696376 ?    Sl   Feb27   3:52      \_ /opt/java/jdk/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -server -XX:+UseParallelGC -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
root      4980  0.0  0.0  65520  2788 ?        Ss   Feb09   0:12 /usr/sbin/sshd -D
root      4346  0.0  0.0  95396  6544 ?        Ss   00:03   0:00  \_ sshd: greenbuds [priv]
greenbu+  4369  0.0  0.0  95396  3468 ?        S    00:03   0:00  |   \_ sshd: greenbuds@pts/8
greenbu+  4370  0.0  0.0  22420  4980 pts/8    Ss   00:03   0:00  |       \_ -bash
greenbu+  4477  0.0  0.0  37676  3696 pts/8    R+   00:05   0:00  |           \_ ps fauxww
root      4394  0.0  0.0  95396  6768 ?        Ss   00:04   0:00  \_ sshd: greenbuds [priv]
greenbu+  4411  0.0  0.0  95396  3288 ?        S    00:04   0:00      \_ sshd: greenbuds@notty
greenbu+  4412  0.0  0.0  12884  1952 ?        Ss   00:04   0:00          \_ /usr/lib/openssh/sftp-server
root     15138  0.0  0.0 148172  1412 ?        Ss   Feb09   0:00 [lxc monitor] /var/lib/lxd/containers qa1
165536   15164  0.0  0.0  41184  2460 ?        Ss   Feb09   0:16  \_ /sbin/init
165536   15215  0.0  0.0  45020 13848 ?        Ss   Feb09   0:07      \_ /usr/lib/systemd/systemd-journald
165536   15216  0.0  0.0  41392   552 ?        Ss   Feb09   0:00      \_ /usr/lib/systemd/systemd-udevd
165617   15239  0.0  0.0  26472   192 ?        Ss   Feb09   0:02      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536   15242  0.0  0.0 295620  7416 ?        Ssl  Feb09   0:01      \_ /usr/sbin/rsyslogd -n
165536   15248  0.0  0.0  26540  1204 ?        Ss   Feb09   0:05      \_ /usr/lib/systemd/systemd-logind
165536   15258  0.0  0.0  22760   616 ?        Ss   Feb09   0:03      \_ /usr/sbin/crond -n
165536   15259  0.0  0.0   6464   120 pts/21   Ss+  Feb09   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165563   15333  0.0  1.5 1992820 380104 ?      Sl   Feb09  25:31      \_ /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
165536   15528  0.0  0.0 110536 13128 ?        Ss   Feb09   0:01      \_ /sbin/dhclient -H qa1 -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165536   26171  0.4 11.5 12433292 2855964 ?    Sl   Feb24  20:06      \_ /opt/java/jdk/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -server -XX:+UseParallelGC -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
root      1337  5.5 21.1 9652600 5232296 ?     Ssl  Feb14 1070:37 /usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
root      8176  0.0  0.0 161520  3008 ?        Sl   Feb14   0:01  \_ /usr/bin/lxd forkexec greenbuds-chat /var/lib/lxd/containers /var/log/lxd/greenbuds-chat/lxc.conf -- env HOME=/root TERM=xterm USER=root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -- cmd bash
165536    8182  0.0  0.0  11776   400 pts/14   Ss+  Feb14   0:00  |   \_ bash
root     20295  0.0  0.0 161264  3436 ?        Sl   Feb15   0:01  \_ /usr/bin/lxd forkexec qa1 /var/lib/lxd/containers /var/log/lxd/qa1/lxc.conf -- env TERM=xterm USER=root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root -- cmd bash
165536   20313  0.0  0.0  11784   700 pts/16   Ss+  Feb15   0:00  |   \_ bash
root     19065  0.0  0.0 161520  5696 ?        Sl   Feb16   0:01  \_ /usr/bin/lxd forkexec greenbuds-invoice /var/lib/lxd/containers /var/log/lxd/greenbuds-invoice/lxc.conf -- env HOME=/root TERM=xterm-256color USER=root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -- cmd bash
165536   19071  0.0  0.0  22392  5620 pts/17   Ss+  Feb16   0:00  |   \_ bash
root     19377  0.0  0.0 162928  3808 ?        Sl   Feb17   0:00  \_ /usr/bin/lxd forkexec qa1 /var/lib/lxd/containers /var/log/lxd/qa1/lxc.conf -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root TERM=xterm-256color USER=root -- cmd bash
165536   19383  0.0  0.0  11784   672 pts/15   Ss+  Feb17   0:00  |   \_ bash
root     20756  0.0  0.0 161776  4060 ?        Sl   Feb17   0:00  \_ /usr/bin/lxd forkexec qa1 /var/lib/lxd/containers /var/log/lxd/qa1/lxc.conf -- env TERM=xterm USER=root HOME=/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -- cmd bash
165536   20762  0.0  0.0  11784   704 pts/18   Ss+  Feb17   0:00  |   \_ bash
root     16427  0.0  0.0 161264  9744 ?        Sl   Feb20   0:00  \_ /usr/bin/lxd forkexec greenbuds-invoice /var/lib/lxd/containers /var/log/lxd/greenbuds-invoice/lxc.conf -- env HOME=/root TERM=xterm-256color USER=root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -- cmd bash
165536   16433  0.0  0.0  22400  6376 pts/22   Ss+  Feb20   0:00  |   \_ bash
root     16804  0.0  0.0 161520 11728 ?        Sl   Feb21   0:00  \_ /usr/bin/lxd forkexec greenbuds-invoice /var/lib/lxd/containers /var/log/lxd/greenbuds-invoice/lxc.conf -- env HOME=/root TERM=xterm-256color USER=root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -- cmd bash
165536   16810  0.0  0.0  22392  6488 pts/4    Ss+  Feb21   0:00  |   \_ bash
root     15178  0.0  0.0 161520  9740 ?        Sl   Feb23   0:00  \_ /usr/bin/lxd forkexec improwise-dev /var/lib/lxd/containers /var/log/lxd/improwise-dev/lxc.conf -- env TERM=xterm-256color USER=root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root -- cmd bash
165536   15184  0.0  0.0  11784  2008 pts/12   Ss+  Feb23   0:00  |   \_ bash
root     23528  0.0  0.0 161520  9752 ?        Sl   Feb23   0:00  \_ /usr/bin/lxd forkexec improwise-dev /var/lib/lxd/containers /var/log/lxd/improwise-dev/lxc.conf -- env HOME=/root TERM=xterm-256color USER=root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -- cmd bash
165536   23534  0.0  0.0  11784  1880 pts/25   Ss+  Feb23   0:00  |   \_ bash
root       834  0.0  0.0 161264  9756 ?        Sl   Feb24   0:00  \_ /usr/bin/lxd forkexec improwise-dev /var/lib/lxd/containers /var/log/lxd/improwise-dev/lxc.conf -- env USER=root HOME=/root TERM=xterm-256color PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -- cmd bash
165536     840  0.0  0.0  11784  1940 pts/23   Ss+  Feb24   0:00      \_ bash
root      7255  0.0  0.0 169968  3792 ?        Ss   Feb14   0:00 [lxc monitor] /var/lib/lxd/containers greenbuds-chat
165536    7279  0.0  0.0  43032  1592 ?        Ss   Feb14   0:03  \_ /sbin/init
165536    7334  0.0  0.0  36828  6144 ?        Ss   Feb14   0:02      \_ /usr/lib/systemd/systemd-journald
165536    7339  0.0  0.0  41400   424 ?        Ss   Feb14   0:00      \_ /usr/lib/systemd/systemd-udevd
165536    7362  0.0  0.0  24200   196 ?        Ss   Feb14   0:00      \_ /usr/lib/systemd/systemd-logind
165536    7364  0.0  0.0 221916    48 ?        Ssl  Feb14   0:00      \_ /usr/sbin/rsyslogd -n
165617    7365  0.0  0.0  24408   184 ?        Ss   Feb14   0:00      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536    7396  0.0  0.0   6456   128 pts/9    Ss+  Feb14   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165563    7429  0.0  0.0   9516    20 ?        Ss   Feb14   0:00      \_ /bin/sh /usr/bin/mysqld_safe --basedir=/usr
165563    7699  0.1  0.7 1580284 195076 ?      Sl   Feb14  27:14      |   \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
165536    7846  0.0  0.0 112832 10596 ?        Ss   Feb14   0:01      \_ /sbin/dhclient -H greenbuds-chat -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
165538   11941  0.2  2.2 12100648 544932 ?     Sl   Feb14  51:31      \_ /opt/openfire/jre/bin/java -server -DopenfireHome=/opt/openfire -Dopenfire.lib.dir=/opt/openfire/lib -classpath /opt/openfire/lib/startup.jar -jar /opt/openfire/lib/startup.jar
root     26211  0.0  0.0 125520  2064 ?        Ss   Feb16   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 26212  0.0  0.0 126692  4788 ?        S    Feb16   1:42  \_ nginx: worker process
www-data 26213  0.0  0.0 126820  4604 ?        S    Feb16   1:40  \_ nginx: worker process
www-data 26214  0.0  0.0 126820  4720 ?        S    Feb16   1:35  \_ nginx: worker process
www-data 26215  0.0  0.0 126436  4516 ?        S    Feb16   1:40  \_ nginx: worker process
www-data 26216  0.0  0.0 126444  4472 ?        S    Feb16   1:42  \_ nginx: worker process
www-data 26217  0.0  0.0 126692  4684 ?        S    Feb16   1:37  \_ nginx: worker process
www-data 26218  0.0  0.0 126692  4352 ?        S    Feb16   1:38  \_ nginx: worker process
www-data 26219  0.0  0.0 126692  4612 ?        S    Feb16   1:40  \_ nginx: worker process
root     19367  0.0  0.0 169712  7340 ?        Ss   Feb27   0:00 [lxc monitor] /var/lib/lxd/containers gimboard-montessorie
165536   19393  0.0  0.0  42968  3556 ?        Ss   Feb27   0:00  \_ /sbin/init
165536   19478  0.0  0.0  36828  5804 ?        Ss   Feb27   0:00      \_ /usr/lib/systemd/systemd-journald
165536   19482  0.0  0.0  41400   400 ?        Ss   Feb27   0:00      \_ /usr/lib/systemd/systemd-udevd
165536   19505  0.0  0.0 221916  2848 ?        Ssl  Feb27   0:00      \_ /usr/sbin/rsyslogd -n
165536   19507  0.0  0.0  24200  2048 ?        Ss   Feb27   0:00      \_ /usr/lib/systemd/systemd-logind
165617   19510  0.0  0.0  24392   836 ?        Ss   Feb27   0:00      \_ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
165536   19541  0.0  0.0  22636  1920 ?        Ss   Feb27   0:00      \_ /usr/sbin/crond -n
165536   19542  0.0  0.0   6456   152 pts/6    Ss+  Feb27   0:00      \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
165536   19547  1.5  5.2 6669236 1300108 ?     Sl   Feb27  11:21      \_ /srv/softwares/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -server -XX:+UseParallelGC -classpath /srv/softwares/tomcat/bin/bootstrap.jar:/srv/softwares/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/srv/softwares/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
165563   19582  0.0  0.0   9516   280 ?        Ss   Feb27   0:00      \_ /bin/sh /usr/bin/mysqld_safe --basedir=/usr
165563   19862  0.3  1.0 4824320 264700 ?      Sl   Feb27   2:27      |   \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
165536   19966  0.0  0.0 112832 13380 ?        Ss   Feb27   0:00      \_ /sbin/dhclient -H gimboard-montessorie -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0

stgraber commented 7 years ago

Not seeing anything obvious. You have a bunch of bash shells running there but I don't see why those would trigger all those events.

@brauner any idea why we seem to be stuck in a "Detected poll(POLLIN | POLLPRI | POLLHUP | POLLRDHUP) event." loop here?

stgraber commented 7 years ago

I'm pretty sure that if you restart the LXD daemon with "sudo systemctl restart lxd" the problem will go away, but that would make it much harder for us to track this down then. So if you can live with the extra load for a bit, it'd be useful for debug purposes.

vasukolli commented 7 years ago

yes, I will leave it as is for a while.

brauner commented 7 years ago

Well, some process is spamming stdout. What are all these forkexec's you have still running? Are this all shells you keep active in the container?

brauner commented 7 years ago

It seems that forkexec processes are left behind although lxc exec has already exited.

stgraber commented 7 years ago

@brauner yeah, I've seen such leftover processes before but they don't usually cause that kind of event loop...

An example to reproduce the leftover processes is:

stgraber@dakara:~$ lxc exec nesting bash
root@nesting:~# Killed
stgraber@dakara:~$ ps aux | grep forkexec
root     29053  0.0  0.0 160064 11596 pts/7    Sl+  14:25   0:00 /home/stgraber/data/code/go/bin/lxd forkexec nesting /var/lib/lxd/containers /var/log/lxd/nesting/lxc.conf -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin LANG=C.UTF-8 HOME=/root TERM=xterm USER=root -- cmd bash
stgraber 29200  0.0  0.0  12960   872 pts/10   S+   14:25   0:00 grep forkexec
stgraber@dakara:~$

So just send SIGKILL to the lxc process (or close the terminal) and you'll have a leftover forkexec. But in my case, the process is happily running, no loop, no wasted CPU, ...

brauner commented 7 years ago

Well, what if you have placed a process in the background that spams stdout...

vasukolli commented 7 years ago

i will not keepthem active.

for qa1 and improwise-dev containers , jenkins from another machine will connect to host machine and will execute "lxc exec container systemctl start/stop " service. May be during this process bash shells persisted

Or

After logging in to container from host machine closing the putty session without logging out from container may caused the persisted bash session.

brauner commented 7 years ago

@stgraber, like yes & or something...

stgraber commented 7 years ago

@brauner I'm not against killing forkexec processes when the connection with the client is lost, however won't that regress your fix for backgrounded processes?

brauner commented 7 years ago

Yeah, very likely. I'd like to avoid this actually. If someone wants to have yes & spamming {her,his} cpu then whatever. We can't really prevent this without breaking legitimate use-cases.

stgraber commented 7 years ago

@brauner Can we differentiate between:

1) Spawn shell (lxc exec bash), then "yes &", then ctrl+d 2) Spawn shell (lxc exec bash), server looses connection with the server

That is, can we differentiate between a user initiated disconnection vs network initiated disconnection?

brauner commented 7 years ago

Hm, good question; need to think about that. But even if the client looses connection with the server: If the user placed a program in the background shouldn't it still be left running?

stgraber commented 7 years ago

@brauner sure but I'm fine with that. If the user does: 1) lxc exec blah bash 2) yes& 3) kill -9 of "lxc exec" process

Then I'd expect LXD to notice a network disconnection from the client, kill the attached process (bash) and forkexec will exit. Leaving the "yes&" running which is fine.

brauner commented 7 years ago

Ah, got it.

mistoo commented 7 years ago

Just tried to reproduce this, and it seems to be reproducible: $ lxc exec testburn bash and then I killed ssh session (it's all on remote machine). Bash is still alive and lxd consumes 100% cpu.

ps: root 21951 0.0 0.1 223120 4520 ? S 20:28 0:00 su - root 21952 0.0 0.0 4560 1648 ? S 20:28 0:00 _ -sh root 21972 0.0 0.1 324404 7796 ? Sl 20:29 0:00 _ lxc exec testburn bash

brauner commented 7 years ago

This should be addressed in https://github.com/lxc/lxd/pull/2968 .

psirek commented 7 years ago

Is this really fixed? I am running LXD 2.13 and I still get "100%" CPU usage by

SSH into the LXD host
"LXC exec container bash"
Close the tab / terminal to terminate the connection

After the above I see much higher CPU consumption. If I do a "LXC exec container bash" again the CPU usage is back to normal. At the moment I am quickfixing this problem by having a CRON job executing a command on all running containers every 5 minutes which also removes the high CPU usage.

brauner commented 7 years ago

What exactly are you terminating how?

brauner commented 7 years ago

Also can you provide the logging output Stéphane requested above, please?

brauner commented 7 years ago

So essentially lxc monitor --type=logging. :)

psirek commented 7 years ago

Hi Brauner

I am doing the following:

SSH nuc.serek.eu
lxc exec nextcloud bash
"Close the iTerm tab from my client that I used to SSH into my server - nuc.serek.eu

Afterward I see this in htop:

If I do this a few times it will hit 100% CPU. Restarting LXD or executing command 2 again will drop the CPU usage to less than 2% on all cores.

lxc monitor output: metadata: context: {} level: dbug message: 'New events listener: 83739bd0-ee3d-4317-a1c2-3659f1316000' timestamp: 2017-06-09T14:11:18.071448881+02:00 type: logging

metadata: context: ip: '@' method: GET url: /1.0 level: dbug message: handling timestamp: 2017-06-09T14:11:26.980976589+02:00 type: logging

metadata: context: {} level: dbug message: Initializing a DIR driver. timestamp: 2017-06-09T14:11:26.982036746+02:00 type: logging

metadata: context: ip: '@' method: POST url: /1.0/containers/nextcloud/exec level: dbug message: handling timestamp: 2017-06-09T14:11:26.984646385+02:00 type: logging

metadata: context: {} level: dbug message: 'New websocket operation: 7d098ea9-2330-4621-86a5-8fb692e54def' timestamp: 2017-06-09T14:11:26.991161631+02:00 type: logging

metadata: context: {} level: dbug message: 'Started websocket operation: 7d098ea9-2330-4621-86a5-8fb692e54def' timestamp: 2017-06-09T14:11:26.99123716+02:00 type: logging

metadata: context: {} level: dbug message: 'Connected websocket operation: 7d098ea9-2330-4621-86a5-8fb692e54def' timestamp: 2017-06-09T14:11:26.993434731+02:00 type: logging

metadata: context: ip: '@' method: GET url: /1.0/operations/7d098ea9-2330-4621-86a5-8fb692e54def/websocket?secret=34ece54d27b168f901028fd407d7e10eb83bfc0cf72aea21b8d95c35ad3fb632 level: dbug message: handling timestamp: 2017-06-09T14:11:26.99339506+02:00 type: logging

metadata: context: {} level: dbug message: 'Handled websocket operation: 7d098ea9-2330-4621-86a5-8fb692e54def' timestamp: 2017-06-09T14:11:26.993528988+02:00 type: logging

metadata: context: ip: '@' method: GET url: /1.0/operations/7d098ea9-2330-4621-86a5-8fb692e54def/websocket?secret=0361aa9d518298a6904d3515b67e2ca360911f1df5fa0ca87aea593242dc153c level: dbug message: handling timestamp: 2017-06-09T14:11:26.995197638+02:00 type: logging

metadata: context: {} level: dbug message: 'Connected websocket operation: 7d098ea9-2330-4621-86a5-8fb692e54def' timestamp: 2017-06-09T14:11:26.995234171+02:00 type: logging

metadata: context: {} level: dbug message: 'Handled websocket operation: 7d098ea9-2330-4621-86a5-8fb692e54def' timestamp: 2017-06-09T14:11:26.995298243+02:00 type: logging

metadata: context: {} level: dbug message: Got message barrier, resetting stream timestamp: 2017-06-09T14:11:41.222015736+02:00 type: logging

metadata: context: {} level: dbug message: Forwarded signal '1' to PID 26131. timestamp: 2017-06-09T14:11:41.224101537+02:00 type: logging

metadata: context: {} level: dbug message: Forwarded signal '18' to PID 26131. timestamp: 2017-06-09T14:11:41.224528462+02:00 type: logging

metadata: context: {} level: dbug message: Forwarded signal '1' to PID 26131. timestamp: 2017-06-09T14:11:41.224350136+02:00 type: logging

metadata: context: {} level: dbug message: 'Got error getting next reader read unix /var/lib/lxd/unix.socket->@: use of closed network connection' timestamp: 2017-06-09T14:11:41.226062103+02:00 type: logging

metadata: context: {} level: dbug message: 'Got error getting next reader read unix /var/lib/lxd/unix.socket->@: use of closed network connection' timestamp: 2017-06-09T14:11:50.1933623+02:00 type: logging

metadata: context: {} level: dbug message: 'Detected poll(POLLHUP) event: exiting.' timestamp: 2017-06-09T14:11:50.193336314+02:00 type: logging

metadata: context: {} level: dbug message: sending write barrier timestamp: 2017-06-09T14:11:50.193956762+02:00 type: logging

metadata: context: ip: '@' method: GET url: /1.0/operations/615fe6d9-59e1-492c-ab7f-967caa0eb6b9/wait level: dbug message: handling timestamp: 2017-06-09T14:11:50.194489772+02:00 type: logging

metadata: context: {} level: dbug message: 'Got error getting next reader websocket: close 1006 (abnormal closure): unexpected EOF, &{%!s(*os.file=&{-1 master })}' timestamp: 2017-06-09T14:11:50.194377427+02:00 type: logging

metadata: context: {} level: dbug message: 'Updated metadata for websocket operation: 615fe6d9-59e1-492c-ab7f-967caa0eb6b9' timestamp: 2017-06-09T14:11:50.194437777+02:00 type: logging

metadata: context: {} level: dbug message: 'Success for websocket operation: 615fe6d9-59e1-492c-ab7f-967caa0eb6b9' timestamp: 2017-06-09T14:11:50.194545885+02:00 type: logging

anoxi commented 7 years ago

SSH into the LXD host "LXC exec container bash" Close the tab / terminal to terminate the connection Efter the above I see much higher CPU consumption. If I do a "LXC exec container bash" again the CPU usage is back to normal. At the moment I am quickfixing this problem by having a CRON job executing a command on all running containers every 5 minutes which also removes the high CPU usage.

I can confirm this behaviour on Ubuntu 16.10 host (lxd 2.12) and Ubuntu 14.04 guest. Workaround for me is now to leave lxd via "exit" on the terminal.

brauner commented 7 years ago

So this is tricky (@stgraber, @hallyn you might find this interesting as well). A while back I implemented signal forwarding for lxc exec, meaning you can send a signal to lxc exec and it will forward it to the program it is executing. So essentially, when you exit the shell lxc exec will receive SIGHUP and forward it to the program that is executing which also means that lxc exec itself will not catch the signal and thus stick around even though the shell session has already exited causing the high cpu load. I need to think about what is the best way to handle this. The good news is that it has nothing to do with the poll()ing code I wrote. That code seems to be stable.

brauner commented 7 years ago

So the simple solution is to simply forward the signal and then send it to ourselves but that somewhat defeats the purpose of the signal-forwarder. The more clever way would be to detect whether our parent has died and then kill ourselves (I know, that's a dramatic statement.).

stgraber commented 7 years ago

Yeah, I suspect for some signals at least we'll want to forward them and then interpret them ourselves in the client.

brauner commented 7 years ago

So yes, that sounds reasonable. I guess SIGHUP is the most obvious candidate here.

hallyn commented 7 years ago

Not being particularly helpful, but just pointing out that this kind of code has been 'delicate' in su :) https://github.com/shadow-maint/shadow/blob/master/src/su.c#L319 Of course in su there's the problem of less-privileged users signaling more-privileged code, which isn't relevant here. So yeah I think you just want to special-case based on signal here.

hallyn commented 7 years ago

Wrong button.

brauner commented 7 years ago

So this is not caused by the SIGHUP I can reproduce this without a signal handler for SIGHUP.

stamster commented 7 years ago

I'm also having this issue. At least I was able to reproduce steps with the community here. It's odd that simple broken connection from parent (host) machine would cause 100% spike on a container. Using lxd/lxc version 2.0.10 (default) on Ubuntu Xerus LTS.

psirek commented 7 years ago

For me this has been fixed on 2.15+, but you need to install the newest version on Ubuntu 16.04. You could do as me and just use a PPA like I have described here: https://odd-one-out.serek.eu/latest-stable-lxd-ubuntu-16-04-lts/ , but I did it on a clean install so you might have less success upgrading from an existing installation.

stgraber commented 7 years ago

Yes, 2.15 fixes it. The other people in this thread are using the LTS branch (2.0.x) and so need to wait for 2.0.11 for this fix to be included.

I wouldn't recommend upgrading from the LTS branch to the latest feature release just for this as there's no way back once you upgrade and there are usually good reasons why you chose the LTS branch in the first place :)

stamster commented 6 years ago

At last - 2.0.11 has been released. https://launchpad.net/ubuntu/+source/lxd/2.0.11-0ubuntu1~16.04.2 I don't see any references to this bug? I hope it is included though... will test it overnight and simulate loosing connection when it goes crazy.

stamster commented 6 years ago

Fixed! My CPU is steady at last! After broken pipe on that remote host... so thumbs up for @stgraber for fixing this in Ubuntu LTS release.

canonical / lxd

lxd high cpu usage #2964

Required information

Issue description

top -c -p 1337

Steps to reproduce

Information to attach

lxc monitor --type=logging output

ps fauxww