search

lxc / lxcfs

FUSE filesystem for LXC
https://linuxcontainers.org/lxcfs
Other
1.04k stars 251 forks source link

LXCFS: crash and Transport endpoint is not connected #73

Closed aarnaud closed 8 years ago

aarnaud commented 8 years ago

Hi,

Often, LXCFS crash and my container i have Transport endpoint is not connected: '/proc/cpuinfo'

I have this bug with Debian Jessie, but not with Ubuntu Trusty container.

Log LXCFS :

sudo cat /var/log/upstart/lxcfs.log
hierarchies: 2: cpuset
 3: cpu
 4: cpuacct
 5: memory
 6: devices
 7: freezer
 8: blkio
 9: hugetlb
 10: perf_event
 11: name=systemd
send_creds: failed at sendmsg: No such process
send_creds: failed at sendmsg: No such process
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
hierarchies: 2: cpuset
 3: cpu
 4: cpuacct
 5: memory
 6: devices
 7: freezer
 8: blkio
 9: hugetlb
 10: perf_event
 11: name=systemd
send_creds: Error getting reply from server over socketpair
hierarchies: 2: cpuset
 3: cpu
 4: blkio
 5: cpuacct
 6: devices
 7: freezer
 8: hugetlb
 9: memory
 10: perf_event
 11: name=systemd
send_creds: failed at sendmsg: No such process
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
Timed out waiting for scm_cred: No such file or directory
send_creds: failed at sendmsg: No such process
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
Timed out waiting for scm_cred: No such file or directory
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
do_read_pids: failed to ask child to exit: No such process
Timed out waiting for scm_cred: No such file or directory
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
Timed out waiting for scm_cred: No such file or directory
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
Timed out waiting for scm_cred: No such file or directory
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
Timed out waiting for scm_cred: No such file or directory
send_creds: failed at sendmsg: No such process
send_creds: failed at sendmsg: No such process
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
Timed out waiting for scm_cred: No such file or directory
send_creds: Error getting reply from server over socketpair
send_creds: failed at sendmsg: No such process
send_creds: failed at sendmsg: No such process
send_creds: failed at sendmsg: No such process
send_creds: failed at sendmsg: No such process
Timed out waiting for scm_cred: Success
send_creds: Error getting reply from server over socketpair
hierarchies: 2: cpuset
 3: cpu
 4: cpuacct
 5: memory
 6: devices
 7: freezer
 8: blkio
 9: perf_event
 10: hugetlb
 11: name=systemd
*** Error in `/usr/bin/lxcfs': double free or corruption (out): 0x00007f58b40009d0 ***
hierarchies: 2: cpuset
 3: cpu
 4: cpuacct
 5: memory
 6: devices
 7: freezer
 8: blkio
 9: perf_event
 10: hugetlb
 11: name=systemd
*** Error in `/usr/bin/lxcfs': munmap_chunk(): invalid pointer: 0x00007f55700010b0 ***
hierarchies: 2: cpuset
 3: cpu
 4: cpuacct
 5: memory
 6: devices
 7: freezer
 8: blkio
 9: perf_event
 10: hugetlb
 11: name=systemd

Versions:

$ lxcfs --version
0.17

$ lxd --version
2.0.0.beta1

$ uname -a
Linux lxc-integcontinue 3.13.0-77-generic #121-Ubuntu SMP  x86_64 x86_64 x86_64 GNU/Linux

Thanks,

hallyn commented 8 years ago

Hi,

so this is a 14.04 host with ubuntu-lxc ppa, and a debian jessie lxd container is causing lxcfs to crash? Does this happen even with a container with nothing really running in it?

Could you try

sudo stop lxcfs sudo gdb lxcfs run /var/lib/lxcfs

then start the container, force the crash, and get a backtrace from gdb?

aarnaud commented 8 years ago

hi,

I use ubuntu 14.04 for the host with lxc-stabe and lxd-stable ppa.

When nothing is running in container, i think, it's ok. But mariadb, php-fpm and nginx services is running in Debian Jessie container.

I try this in company on the host which have the probleme but it's aleatory.

aarnaud commented 8 years ago

First time: 

$ sudo gdb lxcfs
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from lxcfs...(no debugging symbols found)...done.
(gdb) run /var/lib/lxcfs
Starting program: /usr/bin/lxcfs /var/lib/lxcfs
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
hierarchies: 2: cpuset
 3: cpu
 4: cpuacct
 5: memory
 6: blkio
 7: devices
 8: freezer
 9: hugetlb
 10: perf_event
 11: name=systemd
fuse: bad mount point `/var/lib/lxcfs': Transport endpoint is not connected
[Inferior 1 (process 6801) exited with code 01]
(gdb)

I have to reboot host for re-try.

stgraber commented 8 years ago

fusermount -u /var/lib/lxcfs

aarnaud commented 8 years ago

I wait for the crash... With a colleague have it realized that the problem appeared with Debian Jessie 8.3 since December 21

aarnaud commented 8 years ago

Extract of log, but didn't "Transport endpoint" occur. just sometime zombie process (sshd):

[New Thread 0x7fffbbfff700 (LWP 26378)]
[Thread 0x7ffff5bb4700 (LWP 26240) exited]
[New Thread 0x7ffff5bb4700 (LWP 26381)]
[Thread 0x7fffba7fc700 (LWP 25739) exited]
[Thread 0x7fffb97fa700 (LWP 25741) exited]
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
Timed out waiting for scm_cred: Success
[New Thread 0x7fffb97fa700 (LWP 26576)]
[Thread 0x7ffff63b5700 (LWP 25658) exited]
[New Thread 0x7ffff63b5700 (LWP 26633)]
[New Thread 0x7fffba7fc700 (LWP 26640)]
[New Thread 0x7fffdeffd700 (LWP 26642)]
hallyn commented 8 years ago

jessie image isn't happy on my xenial box but lxcfs is not crashing...

Could you still try getting a backtrace in gdb?

hallyn commented 8 years ago

Can you reproduce this on a xenial host?

aarnaud commented 8 years ago

It's difficult to reproduce. this is a random bug. it appear after some days.

send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
Timed out waiting for scm_cred: No such file or directory
send_creds: failed at sendmsg: No such process
send_creds: Error getting reply from server over socketpair
Timed out waiting for scm_cred: No such file or directory

I will try on an xenial host this week if I have time.

aarnaud commented 8 years ago

This time is not with LXD but with LXC :

$ sudo lxc-ls -f
lxc-ls: utils.c: switch_to_ns: 1090 No such file or directory - failed to open /proc/15398/ns/net
lxc-ls: lxccontainer.c: do_lxcapi_get_interfaces: 1796 No such file or directory - failed to enter namespace
lxc-ls: utils.c: switch_to_ns: 1090 No such file or directory - failed to open /proc/15398/ns/net
lxc-ls: lxccontainer.c: do_lxcapi_get_ips: 1888 No such file or directory - failed to enter namespace
lxc-ls: utils.c: switch_to_ns: 1090 No such file or directory - failed to open /proc/15398/ns/net
lxc-ls: lxccontainer.c: do_lxcapi_get_ips: 1888 No such file or directory - failed to enter namespace
aarnaud commented 8 years ago

Host: 

root@inception ~ # lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.3 LTS
Release:    14.04
Codename:   trusty
root@inception ~ # uname -a
Linux inception 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Versions:

root@inception ~ # dpkg -l | grep -E "lxc|lxd"
ii  liblxc1                                                     2.0.0~beta2-0ubuntu2~ubuntu14.04.1~ppa1 amd64        Linux Containers userspace tools (library)
ii  lxc                                                         2.0.0~beta2-0ubuntu2~ubuntu14.04.1~ppa1 amd64        Linux Containers userspace tools
ii  lxc-dev                                                     2.0.0~beta2-0ubuntu2~ubuntu14.04.1~ppa1 amd64        Linux Containers userspace tools (development)
ii  lxc-docker                                                  1.9.1                                   amd64        Linux container runtime
rc  lxc-docker-1.6.2                                            1.6.2                                   amd64        Linux container runtime
rc  lxc-docker-1.7.0                                            1.7.0                                   amd64        Linux container runtime
rc  lxc-docker-1.7.1                                            1.7.1                                   amd64        Linux container runtime
rc  lxc-docker-1.9.0                                            1.9.0                                   amd64        Linux container runtime
ii  lxc-docker-1.9.1                                            1.9.1                                   amd64        Linux container runtime
ii  lxc-templates                                               2.0.0~beta2-0ubuntu2~ubuntu14.04.1~ppa1 amd64        Linux Containers userspace tools (templates)
ii  lxcfs                                                       2.0.0~beta1-0ubuntu1~ubuntu14.04.1~ppa1 amd64        FUSE based filesystem for LXC
ii  lxd                                                         2.0.0~beta2-0ubuntu2~ubuntu14.04.1~ppa1 amd64        Container hypervisor based on LXC - daemon
ii  lxd-client                                                  2.0.0~beta2-0ubuntu2~ubuntu14.04.1~ppa1 amd64        Container hypervisor based on LXC - client
ii  python3-lxc                                                 2.0.0~beta2-0ubuntu2~ubuntu14.04.1~ppa1 amd64        Linux Containers userspace tools (Python 3.x bindings)

Syslog:

Feb 15 05:39:53 inception kernel: [ 2053.040047] type=1400 audit(1455511193.394:197): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=15055 comm="mount" flags="ro, remount, relatime"
Feb 15 05:39:55 inception kernel: [ 2055.057191] vlxcbr1: port 1(veth1pl11342) entered disabled state
Feb 15 05:39:55 inception kernel: [ 2055.088532] vlxcbr1: port 1(veth1pl11342) entered disabled state
Feb 15 05:39:55 inception kernel: [ 2055.088661] lxcbr0: port 3(vethCLI775) entered disabled state
Feb 15 05:39:55 inception kernel: [ 2055.089207] device veth1pl11342 left promiscuous mode
Feb 15 05:39:55 inception kernel: [ 2055.089220] vlxcbr1: port 1(veth1pl11342) entered disabled state
Feb 15 05:39:55 inception kernel: [ 2055.089575] device vethCLI775 left promiscuous mode
Feb 15 05:39:55 inception kernel: [ 2055.089581] lxcbr0: port 3(vethCLI775) entered disabled state
Feb 15 05:40:05 inception kernel: [ 2065.024857] device vethVMI0GO entered promiscuous mode
Feb 15 05:40:05 inception kernel: [ 2065.024949] IPv6: ADDRCONF(NETDEV_UP): vethVMI0GO: link is not ready
Feb 15 05:40:05 inception kernel: [ 2065.140016] IPv6: ADDRCONF(NETDEV_CHANGE): vethVMI0GO: link becomes ready
Feb 15 05:40:05 inception kernel: [ 2065.140049] lxcbr0: port 3(vethVMI0GO) entered forwarding state
Feb 15 05:40:05 inception kernel: [ 2065.140053] lxcbr0: port 3(vethVMI0GO) entered forwarding state
Feb 15 05:40:05 inception kernel: [ 2065.226380] type=1400 audit(1455511205.570:198): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/cgroup/cpu,cpuacct/" pid=15398 comm="systemd" fstype="cgroup" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
Feb 15 05:40:05 inception kernel: [ 2065.226421] type=1400 audit(1455511205.570:199): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/cgroup/" pid=15398 comm="systemd" flags="ro, nosuid, nodev, noexec, remount, strictatime"
Feb 15 05:40:20 inception kernel: [ 2080.163888] lxcbr0: port 3(vethVMI0GO) entered forwarding state
Feb 15 05:40:32 inception kernel: [ 2092.151553] vlxcbr1: port 2(veth1pl30096) entered disabled state
Feb 15 05:40:32 inception kernel: [ 2092.151669] lxcbr0: port 4(vethGR3AJP) entered disabled state
Feb 15 05:40:32 inception kernel: [ 2092.156783] device vethGR3AJP left promiscuous mode
Feb 15 05:40:32 inception kernel: [ 2092.156793] lxcbr0: port 4(vethGR3AJP) entered disabled state
Feb 15 05:38:48 inception snmpd[1506]: message repeated 63 times: [ Connection from UDP: [192.168.0.140]:59948->[192.168.2.41]:161]
Feb 15 05:42:16 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:49317->[192.168.2.41]:161
Feb 15 05:43:42 inception kernel: [ 2282.592711]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 05:43:42 inception kernel: [ 2282.592714] (mount)         D ffff88013fd13180     0 15548  15398 0x00000104
Feb 15 05:43:42 inception kernel: [ 2282.592723]  0000000000013180 0000000000013180 ffff8800a6a13000 ffff88003683c000
Feb 15 05:43:42 inception kernel: [ 2282.592727] Call Trace:
Feb 15 05:43:42 inception kernel: [ 2282.592751]  [<ffffffff812ae305>] __fuse_request_send+0x115/0x280
Feb 15 05:43:42 inception kernel: [ 2282.592758]  [<ffffffff810ab4e0>] ? prepare_to_wait_event+0x100/0x100
Feb 15 05:43:42 inception kernel: [ 2282.592762]  [<ffffffff812b71d7>] fuse_flush+0xd7/0x120
Feb 15 05:43:42 inception kernel: [ 2282.592768]  [<ffffffff811dc33b>] __close_fd+0x7b/0xa0
Feb 15 05:43:42 inception kernel: [ 2282.592770]  [<ffffffff811bbbb3>] SyS_close+0x23/0x50
Feb 15 05:43:42 inception kernel: [ 2282.592773]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 05:42:16 inception snmpd[1506]: message repeated 2 times: [ Connection from UDP: [192.168.0.140]:49317->[192.168.2.41]:161]
Feb 15 05:43:48 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:39673->[192.168.2.41]:161
Feb 15 05:44:38 inception kernel: [ 2338.728166] IPv6: ADDRCONF(NETDEV_CHANGE): veth1pl596: link becomes ready
Feb 15 05:44:38 inception kernel: [ 2338.728210] vlxcbr1: port 1(veth1pl596) entered forwarding state
Feb 15 05:44:38 inception kernel: [ 2338.728215] vlxcbr1: port 1(veth1pl596) entered forwarding state
Feb 15 05:44:41 inception kernel: [ 2341.250458] lxcbr0: port 3(vethVMI0GO) entered disabled state
Feb 15 05:44:56 inception kernel: [ 2356.302073] lxcbr0: port 3(vethVMI0GO) entered forwarding state
Feb 15 05:45:07 inception kernel: [ 2367.018434] lxcbr0: port 3(vethVMI0GO) entered disabled state
Feb 15 05:45:42 inception kernel: [ 2402.666746] INFO: task (mount):15548 blocked for more than 120 seconds.
Feb 15 05:45:42 inception kernel: [ 2402.666752]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 05:45:42 inception kernel: [ 2402.666753] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 05:45:42 inception kernel: [ 2402.666755] (mount)         D ffff88013fd13180     0 15548  15398 0x00000104
Feb 15 05:45:42 inception kernel: [ 2402.666761]  ffff880028f33e30 0000000000000082 ffff8800a6a13000 ffff880028f33fd8
Feb 15 05:45:42 inception kernel: [ 2402.666764]  0000000000013180 0000000000013180 ffff8800a6a13000 ffff88003683c000
Feb 15 05:45:42 inception kernel: [ 2402.666766]  ffff8800b9da8000 ffff880028f33e58 ffff88003683c0e0 ffff8801384c5680
Feb 15 05:45:42 inception kernel: [ 2402.666768] Call Trace:
Feb 15 05:45:42 inception kernel: [ 2402.666793]  [<ffffffff81729549>] schedule+0x29/0x70
Feb 15 05:45:42 inception kernel: [ 2402.666806]  [<ffffffff812ae305>] __fuse_request_send+0x115/0x280
Feb 15 05:45:42 inception kernel: [ 2402.666809]  [<ffffffff812b17d8>] ? fuse_get_req_nofail_nopages+0x98/0x180
Feb 15 05:45:42 inception kernel: [ 2402.666818]  [<ffffffff810ab4e0>] ? prepare_to_wait_event+0x100/0x100
Feb 15 05:45:42 inception kernel: [ 2402.666820]  [<ffffffff812ae482>] fuse_request_send+0x12/0x20
Feb 15 05:45:42 inception kernel: [ 2402.666824]  [<ffffffff812b71d7>] fuse_flush+0xd7/0x120
Feb 15 05:45:42 inception kernel: [ 2402.666839]  [<ffffffff811bbb4f>] filp_close+0x2f/0x70
Feb 15 05:45:42 inception kernel: [ 2402.666846]  [<ffffffff811dc33b>] __close_fd+0x7b/0xa0
Feb 15 05:45:42 inception kernel: [ 2402.666849]  [<ffffffff811bbbb3>] SyS_close+0x23/0x50
Feb 15 05:45:42 inception kernel: [ 2402.666855]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 05:43:48 inception snmpd[1506]: message repeated 63 times: [ Connection from UDP: [192.168.0.140]:39673->[192.168.2.41]:161]
Feb 15 05:46:23 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:56252->[192.168.2.41]:161
Feb 15 05:47:17 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:40216->[192.168.2.41]:161
Feb 15 05:47:42 inception kernel: [ 2522.740748] INFO: task (mount):15548 blocked for more than 120 seconds.
Feb 15 05:47:42 inception kernel: [ 2522.740755]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 05:47:42 inception kernel: [ 2522.740756] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 05:47:42 inception kernel: [ 2522.740758] (mount)         D ffff88013fd13180     0 15548  15398 0x00000104
Feb 15 05:47:42 inception kernel: [ 2522.740763]  ffff880028f33e30 0000000000000082 ffff8800a6a13000 ffff880028f33fd8
Feb 15 05:47:42 inception kernel: [ 2522.740767]  0000000000013180 0000000000013180 ffff8800a6a13000 ffff88003683c000
Feb 15 05:47:42 inception kernel: [ 2522.740770]  ffff8800b9da8000 ffff880028f33e58 ffff88003683c0e0 ffff8801384c5680
Feb 15 05:47:42 inception kernel: [ 2522.740773] Call Trace:
Feb 15 05:47:42 inception kernel: [ 2522.740783]  [<ffffffff81729549>] schedule+0x29/0x70
Feb 15 05:47:42 inception kernel: [ 2522.740789]  [<ffffffff812ae305>] __fuse_request_send+0x115/0x280
Feb 15 05:47:42 inception kernel: [ 2522.740792]  [<ffffffff812b17d8>] ? fuse_get_req_nofail_nopages+0x98/0x180
Feb 15 05:47:42 inception kernel: [ 2522.740795]  [<ffffffff810ab4e0>] ? prepare_to_wait_event+0x100/0x100
Feb 15 05:47:42 inception kernel: [ 2522.740797]  [<ffffffff812ae482>] fuse_request_send+0x12/0x20
Feb 15 05:47:42 inception kernel: [ 2522.740800]  [<ffffffff812b71d7>] fuse_flush+0xd7/0x120
Feb 15 05:47:42 inception kernel: [ 2522.740804]  [<ffffffff811bbb4f>] filp_close+0x2f/0x70
Feb 15 05:47:42 inception kernel: [ 2522.740807]  [<ffffffff811dc33b>] __close_fd+0x7b/0xa0
Feb 15 05:47:42 inception kernel: [ 2522.740808]  [<ffffffff811bbbb3>] SyS_close+0x23/0x50
Feb 15 05:47:42 inception kernel: [ 2522.740812]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 05:47:17 inception snmpd[1506]: message repeated 2 times: [ Connection from UDP: [192.168.0.140]:40216->[192.168.2.41]:161]
Feb 15 05:48:49 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:58185->[192.168.2.41]:161
Feb 15 05:49:42 inception kernel: [ 2642.814810] INFO: task (mount):15548 blocked for more than 120 seconds.
Feb 15 05:49:42 inception kernel: [ 2642.814818]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 05:49:42 inception kernel: [ 2642.814819] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 05:49:42 inception kernel: [ 2642.814821] (mount)         D ffff88013fd13180     0 15548  15398 0x00000104
Feb 15 05:49:42 inception kernel: [ 2642.814825]  ffff880028f33e30 0000000000000082 ffff8800a6a13000 ffff880028f33fd8
Feb 15 05:49:42 inception kernel: [ 2642.814828]  0000000000013180 0000000000013180 ffff8800a6a13000 ffff88003683c000
Feb 15 05:49:42 inception kernel: [ 2642.814829]  ffff8800b9da8000 ffff880028f33e58 ffff88003683c0e0 ffff8801384c5680
Feb 15 05:49:42 inception kernel: [ 2642.814832] Call Trace:
Feb 15 05:49:42 inception kernel: [ 2642.814843]  [<ffffffff81729549>] schedule+0x29/0x70
Feb 15 05:49:42 inception kernel: [ 2642.814849]  [<ffffffff812ae305>] __fuse_request_send+0x115/0x280
Feb 15 05:49:42 inception kernel: [ 2642.814852]  [<ffffffff812b17d8>] ? fuse_get_req_nofail_nopages+0x98/0x180
Feb 15 05:49:42 inception kernel: [ 2642.814855]  [<ffffffff810ab4e0>] ? prepare_to_wait_event+0x100/0x100
Feb 15 05:49:42 inception kernel: [ 2642.814857]  [<ffffffff812ae482>] fuse_request_send+0x12/0x20
Feb 15 05:49:42 inception kernel: [ 2642.814859]  [<ffffffff812b71d7>] fuse_flush+0xd7/0x120
Feb 15 05:49:42 inception kernel: [ 2642.814863]  [<ffffffff811bbb4f>] filp_close+0x2f/0x70
Feb 15 05:49:42 inception kernel: [ 2642.814866]  [<ffffffff811dc33b>] __close_fd+0x7b/0xa0
Feb 15 05:49:42 inception kernel: [ 2642.814867]  [<ffffffff811bbbb3>] SyS_close+0x23/0x50
Feb 15 05:49:42 inception kernel: [ 2642.814870]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 05:51:42 inception kernel: [ 2762.888766] INFO: task (mount):15548 blocked for more than 120 seconds.
Feb 15 05:51:42 inception kernel: [ 2762.888771]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 05:51:42 inception kernel: [ 2762.888772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 05:51:42 inception kernel: [ 2762.888774] (mount)         D ffff88013fd13180     0 15548  15398 0x00000104
Feb 15 05:51:42 inception kernel: [ 2762.888781]  ffff880028f33e30 0000000000000082 ffff8800a6a13000 ffff880028f33fd8
Feb 15 05:51:42 inception kernel: [ 2762.888784]  0000000000013180 0000000000013180 ffff8800a6a13000 ffff88003683c000
Feb 15 05:51:42 inception kernel: [ 2762.888786]  ffff8800b9da8000 ffff880028f33e58 ffff88003683c0e0 ffff8801384c5680
Feb 15 05:51:42 inception kernel: [ 2762.888788] Call Trace:
Feb 15 05:51:42 inception kernel: [ 2762.888796]  [<ffffffff81729549>] schedule+0x29/0x70
Feb 15 05:51:42 inception kernel: [ 2762.888800]  [<ffffffff812ae305>] __fuse_request_send+0x115/0x280
Feb 15 05:51:42 inception kernel: [ 2762.888803]  [<ffffffff812b17d8>] ? fuse_get_req_nofail_nopages+0x98/0x180
Feb 15 05:51:42 inception kernel: [ 2762.888806]  [<ffffffff810ab4e0>] ? prepare_to_wait_event+0x100/0x100
Feb 15 05:51:42 inception kernel: [ 2762.888809]  [<ffffffff812ae482>] fuse_request_send+0x12/0x20
Feb 15 05:51:42 inception kernel: [ 2762.888811]  [<ffffffff812b71d7>] fuse_flush+0xd7/0x120
Feb 15 05:51:42 inception kernel: [ 2762.888815]  [<ffffffff811bbb4f>] filp_close+0x2f/0x70
Feb 15 05:51:42 inception kernel: [ 2762.888817]  [<ffffffff811dc33b>] __close_fd+0x7b/0xa0
Feb 15 05:51:42 inception kernel: [ 2762.888819]  [<ffffffff811bbbb3>] SyS_close+0x23/0x50
Feb 15 05:51:42 inception kernel: [ 2762.888822]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 05:48:49 inception snmpd[1506]: message repeated 63 times: [ Connection from UDP: [192.168.0.140]:58185->[192.168.2.41]:161]
Feb 15 05:52:16 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:48914->[192.168.2.41]:161
Feb 15 05:53:42 inception kernel: [ 2882.962873] INFO: task (mount):15548 blocked for more than 120 seconds.
Feb 15 05:53:42 inception kernel: [ 2882.962881]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 05:53:42 inception kernel: [ 2882.962882] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 05:53:42 inception kernel: [ 2882.962884] (mount)         D ffff88013fd13180     0 15548  15398 0x00000104
Feb 15 05:53:42 inception kernel: [ 2882.962887]  ffff880028f33e30 0000000000000082 ffff8800a6a13000 ffff880028f33fd8
Feb 15 05:53:42 inception kernel: [ 2882.962890]  0000000000013180 0000000000013180 ffff8800a6a13000 ffff88003683c000
Feb 15 05:53:42 inception kernel: [ 2882.962892]  ffff8800b9da8000 ffff880028f33e58 ffff88003683c0e0 ffff8801384c5680
Feb 15 05:53:42 inception kernel: [ 2882.962894] Call Trace:
Feb 15 05:53:42 inception kernel: [ 2882.962902]  [<ffffffff81729549>] schedule+0x29/0x70
Feb 15 05:53:42 inception kernel: [ 2882.962907]  [<ffffffff812ae305>] __fuse_request_send+0x115/0x280
Feb 15 05:53:42 inception kernel: [ 2882.962909]  [<ffffffff812b17d8>] ? fuse_get_req_nofail_nopages+0x98/0x180
Feb 15 05:53:42 inception kernel: [ 2882.962913]  [<ffffffff810ab4e0>] ? prepare_to_wait_event+0x100/0x100
Feb 15 05:53:42 inception kernel: [ 2882.962915]  [<ffffffff812ae482>] fuse_request_send+0x12/0x20
Feb 15 05:53:42 inception kernel: [ 2882.962917]  [<ffffffff812b71d7>] fuse_flush+0xd7/0x120
Feb 15 05:53:42 inception kernel: [ 2882.962921]  [<ffffffff811bbb4f>] filp_close+0x2f/0x70
Feb 15 05:53:42 inception kernel: [ 2882.962924]  [<ffffffff811dc33b>] __close_fd+0x7b/0xa0
Feb 15 05:53:42 inception kernel: [ 2882.962926]  [<ffffffff811bbbb3>] SyS_close+0x23/0x50
Feb 15 05:53:42 inception kernel: [ 2882.962928]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 05:52:16 inception snmpd[1506]: message repeated 2 times: [ Connection from UDP: [192.168.0.140]:48914->[192.168.2.41]:161]
Feb 15 05:53:48 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:32796->[192.168.2.41]:161
Feb 15 05:55:42 inception kernel: [ 3003.036790] INFO: task (mount):15548 blocked for more than 120 seconds.
Feb 15 05:55:42 inception kernel: [ 3003.036798]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 05:55:42 inception kernel: [ 3003.036799] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 05:55:42 inception kernel: [ 3003.036801] (mount)         D ffff88013fd13180     0 15548  15398 0x00000104
Feb 15 05:55:42 inception kernel: [ 3003.036805]  ffff880028f33e30 0000000000000082 ffff8800a6a13000 ffff880028f33fd8
Feb 15 05:55:42 inception kernel: [ 3003.036807]  0000000000013180 0000000000013180 ffff8800a6a13000 ffff88003683c000
Feb 15 05:55:42 inception kernel: [ 3003.036809]  ffff8800b9da8000 ffff880028f33e58 ffff88003683c0e0 ffff8801384c5680
Feb 15 05:55:42 inception kernel: [ 3003.036811] Call Trace:
Feb 15 05:55:42 inception kernel: [ 3003.036825]  [<ffffffff81729549>] schedule+0x29/0x70
Feb 15 05:55:42 inception kernel: [ 3003.036830]  [<ffffffff812ae305>] __fuse_request_send+0x115/0x280
Feb 15 05:55:42 inception kernel: [ 3003.036832]  [<ffffffff812b17d8>] ? fuse_get_req_nofail_nopages+0x98/0x180
Feb 15 05:55:42 inception kernel: [ 3003.036835]  [<ffffffff810ab4e0>] ? prepare_to_wait_event+0x100/0x100
Feb 15 05:55:42 inception kernel: [ 3003.036837]  [<ffffffff812ae482>] fuse_request_send+0x12/0x20
Feb 15 05:55:42 inception kernel: [ 3003.036840]  [<ffffffff812b71d7>] fuse_flush+0xd7/0x120
Feb 15 05:55:42 inception kernel: [ 3003.036844]  [<ffffffff811bbb4f>] filp_close+0x2f/0x70
Feb 15 05:55:42 inception kernel: [ 3003.036846]  [<ffffffff811dc33b>] __close_fd+0x7b/0xa0
Feb 15 05:55:42 inception kernel: [ 3003.036848]  [<ffffffff811bbbb3>] SyS_close+0x23/0x50
Feb 15 05:55:42 inception kernel: [ 3003.036851]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 05:53:48 inception snmpd[1506]: message repeated 63 times: [ Connection from UDP: [192.168.0.140]:32796->[192.168.2.41]:161]
Feb 15 05:56:23 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:54112->[192.168.2.41]:161
Feb 15 05:57:16 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:45850->[192.168.2.41]:161
Feb 15 05:57:42 inception kernel: [ 3123.110728] INFO: task (mount):15548 blocked for more than 120 seconds.
Feb 15 05:57:42 inception kernel: [ 3123.110733]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 05:57:42 inception kernel: [ 3123.110734] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 05:57:42 inception kernel: [ 3123.110736] (mount)         D ffff88013fd13180     0 15548  15398 0x00000104
Feb 15 05:57:42 inception kernel: [ 3123.110740]  ffff880028f33e30 0000000000000082 ffff8800a6a13000 ffff880028f33fd8
Feb 15 05:57:42 inception kernel: [ 3123.110742]  0000000000013180 0000000000013180 ffff8800a6a13000 ffff88003683c000
Feb 15 05:57:42 inception kernel: [ 3123.110744]  ffff8800b9da8000 ffff880028f33e58 ffff88003683c0e0 ffff8801384c5680
Feb 15 05:57:42 inception kernel: [ 3123.110747] Call Trace:
Feb 15 05:57:42 inception kernel: [ 3123.110755]  [<ffffffff81729549>] schedule+0x29/0x70
Feb 15 05:57:42 inception kernel: [ 3123.110759]  [<ffffffff812ae305>] __fuse_request_send+0x115/0x280
Feb 15 05:57:42 inception kernel: [ 3123.110769]  [<ffffffff812b17d8>] ? fuse_get_req_nofail_nopages+0x98/0x180
Feb 15 05:57:42 inception kernel: [ 3123.110772]  [<ffffffff810ab4e0>] ? prepare_to_wait_event+0x100/0x100
Feb 15 05:57:42 inception kernel: [ 3123.110775]  [<ffffffff812ae482>] fuse_request_send+0x12/0x20
Feb 15 05:57:42 inception kernel: [ 3123.110777]  [<ffffffff812b71d7>] fuse_flush+0xd7/0x120
Feb 15 05:57:42 inception kernel: [ 3123.110781]  [<ffffffff811bbb4f>] filp_close+0x2f/0x70
Feb 15 05:57:42 inception kernel: [ 3123.110783]  [<ffffffff811dc33b>] __close_fd+0x7b/0xa0
Feb 15 05:57:42 inception kernel: [ 3123.110785]  [<ffffffff811bbbb3>] SyS_close+0x23/0x50
Feb 15 05:57:42 inception kernel: [ 3123.110788]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 05:57:16 inception snmpd[1506]: message repeated 2 times: [ Connection from UDP: [192.168.0.140]:45850->[192.168.2.41]:161]
Feb 15 05:58:48 inception snmpd[1506]: Connection from UDP: [192.168.0.140]:47360->[192.168.2.41]:161
Feb 15 05:59:42 inception kernel: [ 3243.184832] INFO: task (mount):15548 blocked for more than 120 seconds.
Feb 15 05:59:42 inception kernel: [ 3243.184838]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 05:59:42 inception kernel: [ 3243.184839] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 05:59:42 inception kernel: [ 3243.184840] (mount)         D ffff88013fd13180     0 15548  15398 0x00000104
Feb 15 05:59:42 inception kernel: [ 3243.184844]  ffff880028f33e30 0000000000000082 ffff8800a6a13000 ffff880028f33fd8
Feb 15 05:59:42 inception kernel: [ 3243.184846]  0000000000013180 0000000000013180 ffff8800a6a13000 ffff88003683c000
Feb 15 05:59:42 inception kernel: [ 3243.184848]  ffff8800b9da8000 ffff880028f33e58 ffff88003683c0e0 ffff8801384c5680
Feb 15 05:59:42 inception kernel: [ 3243.184851] Call Trace:
Feb 15 05:59:42 inception kernel: [ 3243.184859]  [<ffffffff81729549>] schedule+0x29/0x70
Feb 15 05:59:42 inception kernel: [ 3243.184863]  [<ffffffff812ae305>] __fuse_request_send+0x115/0x280
Feb 15 05:59:42 inception kernel: [ 3243.184866]  [<ffffffff812b17d8>] ? fuse_get_req_nofail_nopages+0x98/0x180
Feb 15 05:59:42 inception kernel: [ 3243.184869]  [<ffffffff810ab4e0>] ? prepare_to_wait_event+0x100/0x100
Feb 15 05:59:42 inception kernel: [ 3243.184871]  [<ffffffff812ae482>] fuse_request_send+0x12/0x20
Feb 15 05:59:42 inception kernel: [ 3243.184873]  [<ffffffff812b71d7>] fuse_flush+0xd7/0x120
Feb 15 05:59:42 inception kernel: [ 3243.184877]  [<ffffffff811bbb4f>] filp_close+0x2f/0x70
Feb 15 05:59:42 inception kernel: [ 3243.184880]  [<ffffffff811dc33b>] __close_fd+0x7b/0xa0
Feb 15 05:59:42 inception kernel: [ 3243.184881]  [<ffffffff811bbbb3>] SyS_close+0x23/0x50
Feb 15 05:59:42 inception kernel: [ 3243.184884]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6

Thanks

aarnaud commented 8 years ago

On an other Host:

Extract log:

Feb 15 15:13:24 localhost kernel: [85476.075317] init: lxcfs main process (321) killed by ABRT signal
Feb 15 15:13:24 localhost kernel: [85476.075334] init: lxcfs main process ended, respawning

Full logs:

Feb 15 15:05:42 localhost kernel: [85013.585541] INFO: task (agetty):18858 blocked for more than 120 seconds.
Feb 15 15:05:42 localhost kernel: [85013.585638]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:05:42 localhost kernel: [85013.585696] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 15:05:42 localhost kernel: [85013.585750] (agetty)        D ffff88013fc13180     0 18858  12436 0x00000100
Feb 15 15:05:42 localhost kernel: [85013.585754]  ffff88006e5cbc30 0000000000000086 ffff880027109800 ffff88006e5cbfd8
Feb 15 15:05:42 localhost kernel: [85013.585756]  0000000000013180 0000000000013180 ffff880027109800 ffff880035c436a8
Feb 15 15:05:42 localhost kernel: [85013.585758]  ffff880035c436ac ffff880027109800 00000000ffffffff ffff880035c436b0
Feb 15 15:05:42 localhost kernel: [85013.585761] Call Trace:
Feb 15 15:05:42 localhost kernel: [85013.585768]  [<ffffffff81729a69>] schedule_preempt_disabled+0x29/0x70
Feb 15 15:05:42 localhost kernel: [85013.585772]  [<ffffffff8172b8d5>] __mutex_lock_slowpath+0x135/0x1b0
Feb 15 15:05:42 localhost kernel: [85013.585774]  [<ffffffff8172b96f>] mutex_lock+0x1f/0x2f
Feb 15 15:05:42 localhost kernel: [85013.585777]  [<ffffffff81721c20>] lookup_slow+0x33/0xa7
Feb 15 15:05:42 localhost kernel: [85013.585782]  [<ffffffff811cce6a>] link_path_walk+0x83a/0x880
Feb 15 15:05:42 localhost kernel: [85013.585785]  [<ffffffff8131677b>] ? apparmor_file_alloc_security+0x5b/0x180
Feb 15 15:05:42 localhost kernel: [85013.585790]  [<ffffffff812d8d96>] ? security_file_alloc+0x16/0x20
Feb 15 15:05:42 localhost kernel: [85013.585793]  [<ffffffff811cdefc>] path_openat+0x9c/0x640
Feb 15 15:05:42 localhost kernel: [85013.585795]  [<ffffffff811ce53f>] ? getname_flags+0x4f/0x190
Feb 15 15:05:42 localhost kernel: [85013.585797]  [<ffffffff811cf30a>] do_filp_open+0x3a/0x90
Feb 15 15:05:42 localhost kernel: [85013.585800]  [<ffffffff811dc167>] ? __alloc_fd+0xa7/0x130
Feb 15 15:05:42 localhost kernel: [85013.585803]  [<ffffffff811bd8c9>] do_sys_open+0x129/0x280
Feb 15 15:05:42 localhost kernel: [85013.585808]  [<ffffffff810211f7>] ? syscall_trace_enter+0x197/0x250
Feb 15 15:05:42 localhost kernel: [85013.585810]  [<ffffffff811bda3e>] SyS_open+0x1e/0x20
Feb 15 15:05:42 localhost kernel: [85013.585812]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 15:05:42 localhost kernel: [85013.585814] INFO: task (agetty):18956 blocked for more than 120 seconds.
Feb 15 15:05:42 localhost kernel: [85013.585857]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:05:42 localhost kernel: [85013.585892] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 15:05:42 localhost kernel: [85013.585938] (agetty)        D ffff88013fc13180     0 18956  12436 0x00000100
Feb 15 15:05:42 localhost kernel: [85013.585940]  ffff8800061f7c30 0000000000000082 ffff880088601800 ffff8800061f7fd8
Feb 15 15:05:42 localhost kernel: [85013.585942]  0000000000013180 0000000000013180 ffff880088601800 ffff880035c436a8
Feb 15 15:05:42 localhost kernel: [85013.585944]  ffff880035c436ac ffff880088601800 00000000ffffffff ffff880035c436b0
Feb 15 15:05:42 localhost kernel: [85013.585946] Call Trace:
Feb 15 15:05:42 localhost kernel: [85013.585948]  [<ffffffff81729a69>] schedule_preempt_disabled+0x29/0x70
Feb 15 15:05:42 localhost kernel: [85013.585950]  [<ffffffff8172b8d5>] __mutex_lock_slowpath+0x135/0x1b0
Feb 15 15:05:42 localhost kernel: [85013.585952]  [<ffffffff8172b96f>] mutex_lock+0x1f/0x2f
Feb 15 15:05:42 localhost kernel: [85013.585954]  [<ffffffff81721c20>] lookup_slow+0x33/0xa7
Feb 15 15:05:42 localhost kernel: [85013.585957]  [<ffffffff811cce6a>] link_path_walk+0x83a/0x880
Feb 15 15:05:42 localhost kernel: [85013.585959]  [<ffffffff8131677b>] ? apparmor_file_alloc_security+0x5b/0x180
Feb 15 15:05:42 localhost kernel: [85013.585961]  [<ffffffff812d8d96>] ? security_file_alloc+0x16/0x20
Feb 15 15:05:42 localhost kernel: [85013.585963]  [<ffffffff811cdefc>] path_openat+0x9c/0x640
Feb 15 15:05:42 localhost kernel: [85013.585965]  [<ffffffff811ce53f>] ? getname_flags+0x4f/0x190
Feb 15 15:05:42 localhost kernel: [85013.585968]  [<ffffffff811cf30a>] do_filp_open+0x3a/0x90
Feb 15 15:05:42 localhost kernel: [85013.585970]  [<ffffffff811dc167>] ? __alloc_fd+0xa7/0x130
Feb 15 15:05:42 localhost kernel: [85013.585972]  [<ffffffff811bd8c9>] do_sys_open+0x129/0x280
Feb 15 15:05:42 localhost kernel: [85013.585974]  [<ffffffff810211f7>] ? syscall_trace_enter+0x197/0x250
Feb 15 15:05:42 localhost kernel: [85013.585976]  [<ffffffff811bda3e>] SyS_open+0x1e/0x20
Feb 15 15:05:42 localhost kernel: [85013.585978]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 15:05:42 localhost kernel: [85013.585980] INFO: task (agetty):18961 blocked for more than 120 seconds.
Feb 15 15:05:42 localhost kernel: [85013.586020]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:05:42 localhost kernel: [85013.586057] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 15:05:42 localhost kernel: [85013.586100] (agetty)        D ffff88013fc13180     0 18961  12436 0x00000100
Feb 15 15:05:42 localhost kernel: [85013.586102]  ffff8800062d9c30 0000000000000086 ffff880135d78000 ffff8800062d9fd8
Feb 15 15:05:42 localhost kernel: [85013.586104]  0000000000013180 0000000000013180 ffff880135d78000 ffff880035c436a8
Feb 15 15:05:42 localhost kernel: [85013.586106]  ffff880035c436ac ffff880135d78000 00000000ffffffff ffff880035c436b0
Feb 15 15:05:42 localhost kernel: [85013.586108] Call Trace:
Feb 15 15:05:42 localhost kernel: [85013.586110]  [<ffffffff81729a69>] schedule_preempt_disabled+0x29/0x70
Feb 15 15:05:42 localhost kernel: [85013.586112]  [<ffffffff8172b8d5>] __mutex_lock_slowpath+0x135/0x1b0
Feb 15 15:05:42 localhost kernel: [85013.586114]  [<ffffffff8172b96f>] mutex_lock+0x1f/0x2f
Feb 15 15:05:42 localhost kernel: [85013.586117]  [<ffffffff81721c20>] lookup_slow+0x33/0xa7
Feb 15 15:05:42 localhost kernel: [85013.586119]  [<ffffffff811cce6a>] link_path_walk+0x83a/0x880
Feb 15 15:05:42 localhost kernel: [85013.586121]  [<ffffffff8131677b>] ? apparmor_file_alloc_security+0x5b/0x180
Feb 15 15:05:42 localhost kernel: [85013.586125]  [<ffffffff812d8d96>] ? security_file_alloc+0x16/0x20
Feb 15 15:05:42 localhost kernel: [85013.586128]  [<ffffffff811cdefc>] path_openat+0x9c/0x640
Feb 15 15:05:42 localhost kernel: [85013.586130]  [<ffffffff811ce53f>] ? getname_flags+0x4f/0x190
Feb 15 15:05:42 localhost kernel: [85013.586132]  [<ffffffff811cf30a>] do_filp_open+0x3a/0x90
Feb 15 15:05:42 localhost kernel: [85013.586134]  [<ffffffff811dc167>] ? __alloc_fd+0xa7/0x130
Feb 15 15:05:42 localhost kernel: [85013.586136]  [<ffffffff811bd8c9>] do_sys_open+0x129/0x280
Feb 15 15:05:42 localhost kernel: [85013.586139]  [<ffffffff810211f7>] ? syscall_trace_enter+0x197/0x250
Feb 15 15:05:42 localhost kernel: [85013.586141]  [<ffffffff811bda3e>] SyS_open+0x1e/0x20
Feb 15 15:05:42 localhost kernel: [85013.586143]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 15:06:15 localhost kernel: [85046.024792] device vethEW5GAA entered promiscuous mode
Feb 15 15:06:15 localhost kernel: [85046.024891] IPv6: ADDRCONF(NETDEV_UP): vethEW5GAA: link is not ready
Feb 15 15:06:15 localhost kernel: [85046.600601] type=1400 audit(1455545175.616:60): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxd-le-village-by-ca-server-web_</var/lib/lxd>" name="/sys/" pid=21374 comm="mount" flags="rw, nosuid, nodev, noexec, remount"
Feb 15 15:06:15 localhost kernel: [85046.797533] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Feb 15 15:06:16 localhost kernel: [85047.024101] IPv6: ADDRCONF(NETDEV_CHANGE): vethEW5GAA: link becomes ready
Feb 15 15:06:16 localhost kernel: [85047.024125] lxcbr0: port 5(vethEW5GAA) entered forwarding state
Feb 15 15:06:16 localhost kernel: [85047.024130] lxcbr0: port 5(vethEW5GAA) entered forwarding state
Feb 15 15:06:31 localhost kernel: [85062.055312] lxcbr0: port 5(vethEW5GAA) entered forwarding state
Feb 15 15:07:42 localhost kernel: [85133.659631]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:07:42 localhost kernel: [85133.659653] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 15:07:42 localhost kernel: [85133.659682] (agetty)        D ffff88013fc13180     0 18858  12436 0x00000100
Feb 15 15:07:42 localhost kernel: [85133.659686]  ffff88006e5cbc30 0000000000000086 ffff880027109800 ffff88006e5cbfd8
Feb 15 15:07:42 localhost kernel: [85133.659690]  0000000000013180 0000000000013180 ffff880027109800 ffff880035c436a8
Feb 15 15:07:42 localhost kernel: [85133.659694] Call Trace:
Feb 15 15:07:42 localhost kernel: [85133.659720]  [<ffffffff8172b8d5>] __mutex_lock_slowpath+0x135/0x1b0
Feb 15 15:07:42 localhost kernel: [85133.659728]  [<ffffffff81721c20>] lookup_slow+0x33/0xa7
Feb 15 15:07:42 localhost kernel: [85133.659753]  [<ffffffff8131677b>] ? apparmor_file_alloc_security+0x5b/0x180
Feb 15 15:07:42 localhost kernel: [85133.659765]  [<ffffffff811cdefc>] path_openat+0x9c/0x640
Feb 15 15:07:42 localhost kernel: [85133.659770]  [<ffffffff811cf30a>] do_filp_open+0x3a/0x90
Feb 15 15:07:42 localhost kernel: [85133.659777]  [<ffffffff811bd8c9>] do_sys_open+0x129/0x280
Feb 15 15:07:42 localhost kernel: [85133.659793]  [<ffffffff811bda3e>] SyS_open+0x1e/0x20
Feb 15 15:07:42 localhost kernel: [85133.659796]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 15:07:42 localhost kernel: [85133.659798] INFO: task (agetty):18956 blocked for more than 120 seconds.
Feb 15 15:07:42 localhost kernel: [85133.659824]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:07:42 localhost kernel: [85133.659845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 15:07:42 localhost kernel: [85133.659874] (agetty)        D ffff88013fc13180     0 18956  12436 0x00000100
Feb 15 15:07:42 localhost kernel: [85133.659876]  ffff8800061f7c30 0000000000000082 ffff880088601800 ffff8800061f7fd8
Feb 15 15:07:42 localhost kernel: [85133.659879]  0000000000013180 0000000000013180 ffff880088601800 ffff880035c436a8
Feb 15 15:07:42 localhost kernel: [85133.659880]  ffff880035c436ac ffff880088601800 00000000ffffffff ffff880035c436b0
Feb 15 15:07:42 localhost kernel: [85133.659882] Call Trace:
Feb 15 15:07:42 localhost kernel: [85133.659885]  [<ffffffff81729a69>] schedule_preempt_disabled+0x29/0x70
Feb 15 15:07:42 localhost kernel: [85133.659889]  [<ffffffff8172b96f>] mutex_lock+0x1f/0x2f
Feb 15 15:07:42 localhost kernel: [85133.659893]  [<ffffffff811cce6a>] link_path_walk+0x83a/0x880
Feb 15 15:07:42 localhost kernel: [85133.660027]  [<ffffffff812d8d96>] ? security_file_alloc+0x16/0x20
Feb 15 15:07:42 localhost kernel: [85133.660040]  [<ffffffff811ce53f>] ? getname_flags+0x4f/0x190
Feb 15 15:07:42 localhost kernel: [85133.660044]  [<ffffffff811dc167>] ? __alloc_fd+0xa7/0x130
Feb 15 15:07:42 localhost kernel: [85133.660050]  [<ffffffff810211f7>] ? syscall_trace_enter+0x197/0x250
Feb 15 15:07:42 localhost kernel: [85133.660055]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 15:07:42 localhost kernel: [85133.660086]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:07:42 localhost kernel: [85133.660136] (agetty)        D ffff88013fc13180     0 18961  12436 0x00000100
Feb 15 15:07:42 localhost kernel: [85133.660141]  0000000000013180 0000000000013180 ffff880135d78000 ffff880035c436a8
Feb 15 15:07:42 localhost kernel: [85133.660145] Call Trace:
Feb 15 15:07:42 localhost kernel: [85133.660150]  [<ffffffff8172b8d5>] __mutex_lock_slowpath+0x135/0x1b0
Feb 15 15:07:42 localhost kernel: [85133.660155]  [<ffffffff81721c20>] lookup_slow+0x33/0xa7
Feb 15 15:07:42 localhost kernel: [85133.660160]  [<ffffffff8131677b>] ? apparmor_file_alloc_security+0x5b/0x180
Feb 15 15:07:42 localhost kernel: [85133.660165]  [<ffffffff811cdefc>] path_openat+0x9c/0x640
Feb 15 15:07:42 localhost kernel: [85133.660170]  [<ffffffff811cf30a>] do_filp_open+0x3a/0x90
Feb 15 15:07:42 localhost kernel: [85133.660174]  [<ffffffff811bd8c9>] do_sys_open+0x129/0x280
Feb 15 15:07:42 localhost kernel: [85133.660178]  [<ffffffff811bda3e>] SyS_open+0x1e/0x20
Feb 15 15:07:48 localhost kernel: [85139.543359] lxcbr0: port 5(vethEW5GAA) entered disabled state
Feb 15 15:07:48 localhost kernel: [85139.543905] device vethEW5GAA left promiscuous mode
Feb 15 15:07:48 localhost kernel: [85139.543929] lxcbr0: port 5(vethEW5GAA) entered disabled state
Feb 15 15:07:49 localhost kernel: [85140.120323] type=1400 audit(1455545269.080:61): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-le-village-by-ca-server-web_</var/lib/lxd>" pid=25273 comm="apparmor_parser"
Feb 15 15:09:01 localhost CRON[25869]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Feb 15 15:09:42 localhost kernel: [85253.733493] INFO: task (agetty):18858 blocked for more than 120 seconds.
Feb 15 15:09:42 localhost kernel: [85253.733544]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:09:42 localhost kernel: [85253.733573] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 15:09:42 localhost kernel: [85253.733612] (agetty)        D ffff88013fc13180     0 18858  12436 0x00000100
Feb 15 15:09:42 localhost kernel: [85253.733620]  ffff88006e5cbc30 0000000000000086 ffff880027109800 ffff88006e5cbfd8
Feb 15 15:09:42 localhost kernel: [85253.733623]  0000000000013180 0000000000013180 ffff880027109800 ffff880035c436a8
Feb 15 15:09:42 localhost kernel: [85253.733624]  ffff880035c436ac ffff880027109800 00000000ffffffff ffff880035c436b0
Feb 15 15:09:42 localhost kernel: [85253.733627] Call Trace:
Feb 15 15:09:42 localhost kernel: [85253.733636]  [<ffffffff81729a69>] schedule_preempt_disabled+0x29/0x70
Feb 15 15:09:42 localhost kernel: [85253.733639]  [<ffffffff8172b8d5>] __mutex_lock_slowpath+0x135/0x1b0
Feb 15 15:09:42 localhost kernel: [85253.733641]  [<ffffffff8172b96f>] mutex_lock+0x1f/0x2f
Feb 15 15:09:42 localhost kernel: [85253.733644]  [<ffffffff81721c20>] lookup_slow+0x33/0xa7
Feb 15 15:09:42 localhost kernel: [85253.733649]  [<ffffffff811cce6a>] link_path_walk+0x83a/0x880
Feb 15 15:09:42 localhost kernel: [85253.733653]  [<ffffffff8131677b>] ? apparmor_file_alloc_security+0x5b/0x180
Feb 15 15:09:42 localhost kernel: [85253.733658]  [<ffffffff812d8d96>] ? security_file_alloc+0x16/0x20
Feb 15 15:09:42 localhost kernel: [85253.733660]  [<ffffffff811cdefc>] path_openat+0x9c/0x640
Feb 15 15:09:42 localhost kernel: [85253.733663]  [<ffffffff811ce53f>] ? getname_flags+0x4f/0x190
Feb 15 15:09:42 localhost kernel: [85253.733665]  [<ffffffff811cf30a>] do_filp_open+0x3a/0x90
Feb 15 15:09:42 localhost kernel: [85253.733667]  [<ffffffff811dc167>] ? __alloc_fd+0xa7/0x130
Feb 15 15:09:42 localhost kernel: [85253.733671]  [<ffffffff811bd8c9>] do_sys_open+0x129/0x280
Feb 15 15:09:42 localhost kernel: [85253.733676]  [<ffffffff810211f7>] ? syscall_trace_enter+0x197/0x250
Feb 15 15:09:42 localhost kernel: [85253.733678]  [<ffffffff811bda3e>] SyS_open+0x1e/0x20
Feb 15 15:09:42 localhost kernel: [85253.733681]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 15:09:42 localhost kernel: [85253.733683] INFO: task (agetty):18956 blocked for more than 120 seconds.
Feb 15 15:09:42 localhost kernel: [85253.733709]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:09:42 localhost kernel: [85253.733730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 15:09:42 localhost kernel: [85253.733759] (agetty)        D ffff88013fc13180     0 18956  12436 0x00000100
Feb 15 15:09:42 localhost kernel: [85253.733761]  ffff8800061f7c30 0000000000000082 ffff880088601800 ffff8800061f7fd8
Feb 15 15:09:42 localhost kernel: [85253.733763]  0000000000013180 0000000000013180 ffff880088601800 ffff880035c436a8
Feb 15 15:09:42 localhost kernel: [85253.733765]  ffff880035c436ac ffff880088601800 00000000ffffffff ffff880035c436b0
Feb 15 15:09:42 localhost kernel: [85253.733767] Call Trace:
Feb 15 15:09:42 localhost kernel: [85253.733769]  [<ffffffff81729a69>] schedule_preempt_disabled+0x29/0x70
Feb 15 15:09:42 localhost kernel: [85253.733771]  [<ffffffff8172b8d5>] __mutex_lock_slowpath+0x135/0x1b0
Feb 15 15:09:42 localhost kernel: [85253.733773]  [<ffffffff8172b96f>] mutex_lock+0x1f/0x2f
Feb 15 15:09:42 localhost kernel: [85253.733786]  [<ffffffff81721c20>] lookup_slow+0x33/0xa7
Feb 15 15:09:42 localhost kernel: [85253.733789]  [<ffffffff811cce6a>] link_path_walk+0x83a/0x880
Feb 15 15:09:42 localhost kernel: [85253.733791]  [<ffffffff8131677b>] ? apparmor_file_alloc_security+0x5b/0x180
Feb 15 15:09:42 localhost kernel: [85253.733794]  [<ffffffff812d8d96>] ? security_file_alloc+0x16/0x20
Feb 15 15:09:42 localhost kernel: [85253.733796]  [<ffffffff811cdefc>] path_openat+0x9c/0x640
Feb 15 15:09:42 localhost kernel: [85253.733798]  [<ffffffff811ce53f>] ? getname_flags+0x4f/0x190
Feb 15 15:09:42 localhost kernel: [85253.733800]  [<ffffffff811cf30a>] do_filp_open+0x3a/0x90
Feb 15 15:09:42 localhost kernel: [85253.733802]  [<ffffffff811dc167>] ? __alloc_fd+0xa7/0x130
Feb 15 15:09:42 localhost kernel: [85253.733805]  [<ffffffff811bd8c9>] do_sys_open+0x129/0x280
Feb 15 15:09:42 localhost kernel: [85253.733807]  [<ffffffff810211f7>] ? syscall_trace_enter+0x197/0x250
Feb 15 15:09:42 localhost kernel: [85253.733810]  [<ffffffff811bda3e>] SyS_open+0x1e/0x20
Feb 15 15:09:42 localhost kernel: [85253.733813]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 15:09:42 localhost kernel: [85253.733815] INFO: task (agetty):18961 blocked for more than 120 seconds.
Feb 15 15:09:42 localhost kernel: [85253.733846]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:09:42 localhost kernel: [85253.733908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 15:09:42 localhost kernel: [85253.733937] (agetty)        D ffff88013fc13180     0 18961  12436 0x00000100
Feb 15 15:09:42 localhost kernel: [85253.733940]  ffff8800062d9c30 0000000000000086 ffff880135d78000 ffff8800062d9fd8
Feb 15 15:09:42 localhost kernel: [85253.733942]  0000000000013180 0000000000013180 ffff880135d78000 ffff880035c436a8
Feb 15 15:09:42 localhost kernel: [85253.733943]  ffff880035c436ac ffff880135d78000 00000000ffffffff ffff880035c436b0
Feb 15 15:09:42 localhost kernel: [85253.733945] Call Trace:
Feb 15 15:09:42 localhost kernel: [85253.733949]  [<ffffffff81729a69>] schedule_preempt_disabled+0x29/0x70
Feb 15 15:09:42 localhost kernel: [85253.733951]  [<ffffffff8172b8d5>] __mutex_lock_slowpath+0x135/0x1b0
Feb 15 15:09:42 localhost kernel: [85253.733953]  [<ffffffff8172b96f>] mutex_lock+0x1f/0x2f
Feb 15 15:09:42 localhost kernel: [85253.733955]  [<ffffffff81721c20>] lookup_slow+0x33/0xa7
Feb 15 15:09:42 localhost kernel: [85253.733958]  [<ffffffff811cce6a>] link_path_walk+0x83a/0x880
Feb 15 15:09:42 localhost kernel: [85253.733960]  [<ffffffff8131677b>] ? apparmor_file_alloc_security+0x5b/0x180
Feb 15 15:09:42 localhost kernel: [85253.733963]  [<ffffffff812d8d96>] ? security_file_alloc+0x16/0x20
Feb 15 15:09:42 localhost kernel: [85253.733965]  [<ffffffff811cdefc>] path_openat+0x9c/0x640
Feb 15 15:09:42 localhost kernel: [85253.733967]  [<ffffffff811ce53f>] ? getname_flags+0x4f/0x190
Feb 15 15:09:42 localhost kernel: [85253.733969]  [<ffffffff811cf30a>] do_filp_open+0x3a/0x90
Feb 15 15:09:42 localhost kernel: [85253.733971]  [<ffffffff811dc167>] ? __alloc_fd+0xa7/0x130
Feb 15 15:09:42 localhost kernel: [85253.733974]  [<ffffffff811bd8c9>] do_sys_open+0x129/0x280
Feb 15 15:09:42 localhost kernel: [85253.733976]  [<ffffffff810211f7>] ? syscall_trace_enter+0x197/0x250
Feb 15 15:09:42 localhost kernel: [85253.733978]  [<ffffffff811bda3e>] SyS_open+0x1e/0x20
Feb 15 15:09:42 localhost kernel: [85253.733980]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 15:11:42 localhost kernel: [85373.807520] INFO: task (agetty):18858 blocked for more than 120 seconds.
Feb 15 15:11:42 localhost kernel: [85373.807558]       Not tainted 3.13.0-77-generic #121-Ubuntu
Feb 15 15:11:42 localhost kernel: [85373.807580] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 15:11:42 localhost kernel: [85373.807609] (agetty)        D ffff88013fc13180     0 18858  12436 0x00000100
Feb 15 15:11:42 localhost kernel: [85373.807613]  ffff88006e5cbc30 0000000000000086 ffff880027109800 ffff88006e5cbfd8
Feb 15 15:11:42 localhost kernel: [85373.807616]  0000000000013180 0000000000013180 ffff880027109800 ffff880035c436a8
Feb 15 15:11:42 localhost kernel: [85373.807618]  ffff880035c436ac ffff880027109800 00000000ffffffff ffff880035c436b0
Feb 15 15:11:42 localhost kernel: [85373.807620] Call Trace:
Feb 15 15:11:42 localhost kernel: [85373.807629]  [<ffffffff81729a69>] schedule_preempt_disabled+0x29/0x70
Feb 15 15:11:42 localhost kernel: [85373.807632]  [<ffffffff8172b8d5>] __mutex_lock_slowpath+0x135/0x1b0
Feb 15 15:11:42 localhost kernel: [85373.807634]  [<ffffffff8172b96f>] mutex_lock+0x1f/0x2f
Feb 15 15:11:42 localhost kernel: [85373.807638]  [<ffffffff81721c20>] lookup_slow+0x33/0xa7
Feb 15 15:11:42 localhost kernel: [85373.807643]  [<ffffffff811cce6a>] link_path_walk+0x83a/0x880
Feb 15 15:11:42 localhost kernel: [85373.807647]  [<ffffffff8131677b>] ? apparmor_file_alloc_security+0x5b/0x180
Feb 15 15:11:42 localhost kernel: [85373.807652]  [<ffffffff812d8d96>] ? security_file_alloc+0x16/0x20
Feb 15 15:11:42 localhost kernel: [85373.807655]  [<ffffffff811cdefc>] path_openat+0x9c/0x640
Feb 15 15:11:42 localhost kernel: [85373.807657]  [<ffffffff811ce53f>] ? getname_flags+0x4f/0x190
Feb 15 15:11:42 localhost kernel: [85373.807659]  [<ffffffff811cf30a>] do_filp_open+0x3a/0x90
Feb 15 15:11:42 localhost kernel: [85373.807662]  [<ffffffff811dc167>] ? __alloc_fd+0xa7/0x130
Feb 15 15:11:42 localhost kernel: [85373.807666]  [<ffffffff811bd8c9>] do_sys_open+0x129/0x280
Feb 15 15:11:42 localhost kernel: [85373.807671]  [<ffffffff810211f7>] ? syscall_trace_enter+0x197/0x250
Feb 15 15:11:42 localhost kernel: [85373.807673]  [<ffffffff811bda3e>] SyS_open+0x1e/0x20
Feb 15 15:11:42 localhost kernel: [85373.807677]  [<ffffffff81735f2f>] tracesys+0xe1/0xe6
Feb 15 15:13:24 localhost kernel: [85476.075317] init: lxcfs main process (321) killed by ABRT signal
Feb 15 15:13:24 localhost kernel: [85476.075334] init: lxcfs main process ended, respawning
aarnaud commented 8 years ago

I tried with an other kernel : 3.19.0-50-generic since monday, Not bugs

hallyn commented 8 years ago

If you can reproduce this with recent lxcfs, please reopen.

hloeung commented 5 years ago

Ran into this today:

Dec 8 06:25:03 orlo kernel: [25247258.665022] lxcfs[3871]: segfault at 0 ip 00007fe70ae4e3b2 sp 00007fe70884fb70 error 4 in liblxcfs.so[7fe70ae46000+f000] Dec 8 06:25:09 orlo systemd[1]: lxcfs.service: Main process exited, code=killed, status=11/SEGV Dec 8 06:25:09 orlo systemd[1]: lxcfs.service: Unit entered failed state. Dec 8 06:25:09 orlo systemd[1]: lxcfs.service: Failed with result 'signal'. Dec 8 06:25:10 orlo systemd[1]: lxcfs.service: Service hold-off time over, scheduling restart. Dec 8 06:25:10 orlo lxcfs[10839]: hierarchies: Dec 8 06:25:10 orlo lxcfs[10839]: 0: fd: 5: perf_event Dec 8 06:25:10 orlo lxcfs[10839]: 1: fd: 6: blkio Dec 8 06:25:10 orlo lxcfs[10839]: 2: fd: 7: freezer Dec 8 06:25:10 orlo lxcfs[10839]: 3: fd: 8: devices Dec 8 06:25:10 orlo lxcfs[10839]: 4: fd: 9: cpuset Dec 8 06:25:10 orlo lxcfs[10839]: 5: fd: 10: cpu,cpuacct Dec 8 06:25:10 orlo lxcfs[10839]: 6: fd: 11: pids Dec 8 06:25:10 orlo lxcfs[10839]: 7: fd: 12: memory Dec 8 06:25:10 orlo lxcfs[10839]: 8: fd: 13: net_cls,net_prio Dec 8 06:25:10 orlo lxcfs[10839]: 9: fd: 14: hugetlb Dec 8 06:25:10 orlo lxcfs[10839]: 10: fd: 15: name=systemd

So now after restarting it, the containers are showing this:

Error: /proc must be mounted To mount /proc at boot you need an /etc/fstab line like: proc /proc proc defaults In the meantime, run "mount proc /proc -t proc"

What can we do about this?

ubuntu@orlo:~$ cfs lxcfs: Installed: 2.0.8-0ubuntu1~16.04.2 Candidate: 2.0.8-0ubuntu1~16.04.2 Version table: 3.0.2-0ubuntu1~16.04.1 100 100 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages *** 2.0.8-0ubuntu1~16.04.2 500 500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages 100 /var/lib/dpkg/status 2.0.0-0ubuntu2 500 500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

hallyn commented 5 years ago

Please open a new bug, as I don't believe it is related.

In that bug, please show anything interesting from the syslog around that time. The easiest way to debug this would be if you could start lxcfs in gdb, so that when it crashes you could examine the memory around the crashing ip.

hloeung commented 5 years ago

Right, apologies, I filed https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1807628 shortly after and uploaded the .crash/core. Anyways, lets leave this one closed :)