gluster / gluster-kubernetes

GlusterFS Native Storage Service for Kubernetes
Apache License 2.0
875 stars 390 forks source link

Kernel stack traces in log #442

Open paphillon opened 6 years ago

paphillon commented 6 years ago

Container gluster/gluster-centos Glusterfs Version 3.10 Kubernetes Version: 1.9.2 Host OS CentOS 7 Worker nodes: 3

After installing glusterfs I noticed the below kernel stack traces in log. However, the setup is working as expected and I am able to create/delete volumes.

How do I fix this?

os-release

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

uname -a

Linux glusterfs-server-1 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Stack trace-- ` Feb 16 23:01:24 glusterfs-server-1 kernel: IPVS: Creating netns size=2040 id=1 Feb 16 23:01:24 glusterfs-server-1 kernel: weave: port 2(vethwepl9acd7ca) entered blocking state Feb 16 23:01:24 glusterfs-server-1 kernel: weave: port 2(vethwepl9acd7ca) entered disabled state Feb 16 23:01:24 glusterfs-server-1 kernel: device vethwepl9acd7ca entered promiscuous mode Feb 16 23:01:24 glusterfs-server-1 kernel: IPv6: ADDRCONF(NETDEV_UP): vethwepl9acd7ca: link is not ready Feb 16 23:01:24 glusterfs-server-1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethwepl9acd7ca: link becomes ready Feb 16 23:01:24 glusterfs-server-1 kernel: weave: port 2(vethwepl9acd7ca) entered blocking state Feb 16 23:01:24 glusterfs-server-1 kernel: weave: port 2(vethwepl9acd7ca) entered forwarding state Feb 16 23:01:24 glusterfs-server-1 kernel: XFS (dm-12): Mounting V4 Filesystem Feb 16 23:01:24 glusterfs-server-1 kernel: XFS (dm-12): Ending clean mount Feb 16 23:01:24 glusterfs-server-1 kernel: XFS (dm-12): Unmounting Filesystem Feb 16 23:01:24 glusterfs-server-1 kernel: XFS (dm-12): Mounting V4 Filesystem Feb 16 23:01:24 glusterfs-server-1 kernel: XFS (dm-12): Ending clean mount Feb 16 23:01:24 glusterfs-server-1 kernel: XFS (dm-12): Unmounting Filesystem Feb 16 23:01:24 glusterfs-server-1 kernel: XFS (dm-12): Mounting V4 Filesystem Feb 16 23:01:24 glusterfs-server-1 kernel: XFS (dm-12): Ending clean mount Feb 16 23:01:24 glusterfs-server-1 kernel: device-mapper: thin: Data device (dm-14) discard unsupported: Disabling discard passdown. Feb 16 23:01:24 glusterfs-server-1 kernel: ============================================================================= Feb 16 23:01:24 glusterfs-server-1 kernel: BUG kmalloc-256(5:de25949f9c337499646227875f90b6e096531cb23e84bcac17802d355fd3cd05) (Not tainted): Objects remaining in kmalloc-256(5:de25949f9c337499646227875f90b6e096531cb23e84bcac17802d355fd3cd05 Feb 16 23:01:24 glusterfs-server-1 kernel: ----------------------------------------------------------------------------- Feb 16 23:01:24 glusterfs-server-1 kernel: Disabling lock debugging due to kernel taint Feb 16 23:01:24 glusterfs-server-1 kernel: INFO: Slab 0xffffea001062f540 objects=16 used=4 fp=0xffff880418bd5600 flags=0x2fffff00000080 Feb 16 23:01:24 glusterfs-server-1 kernel: CPU: 2 PID: 4911 Comm: lvcreate Tainted: G B ------------ 3.10.0-693.5.2.el7.x86_64 #1 Feb 16 23:01:24 glusterfs-server-1 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 Feb 16 23:01:24 glusterfs-server-1 kernel: ffffea001062f540 0000000069a8e6fa ffff8803ca3f7aa0 ffffffff816a3e51 Feb 16 23:01:24 glusterfs-server-1 kernel: ffff8803ca3f7b78 ffffffff811dbec4 ffff880400000020 ffff8803ca3f7b88 Feb 16 23:01:24 glusterfs-server-1 kernel: ffff8803ca3f7b38 656a624f00000297 616d657220737463 6e6920676e696e69 Feb 16 23:01:24 glusterfs-server-1 kernel: Call Trace: Feb 16 23:01:24 glusterfs-server-1 kernel: [] dump_stack+0x19/0x1b Feb 16 23:01:24 glusterfs-server-1 kernel: [] slab_err+0xb4/0xe0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? kmalloc+0x1e3/0x230 Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? kmem_cache_close+0x127/0x2e0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] kmem_cache_close+0x149/0x2e0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] kmem_cache_shutdown+0x14/0x80 Feb 16 23:01:24 glusterfs-server-1 kernel: [] kmem_cache_destroy+0x44/0xf0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] kmem_cache_destroy_memcg_children+0x89/0xb0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] kmem_cache_destroy+0x19/0xf0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] bioset_free+0xce/0x110 Feb 16 23:01:24 glusterfs-server-1 kernel: [] dm_destroy+0x1ae/0x310 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] dm_destroy+0x13/0x20 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] dev_remove+0x11e/0x180 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? dev_suspend+0x250/0x250 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] ctl_ioctl+0x1e5/0x500 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] dm_ctl_ioctl+0x13/0x20 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] do_vfs_ioctl+0x33d/0x540 Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? file_has_perm+0x9f/0xb0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? __fput+0xe/0x10 Feb 16 23:01:24 glusterfs-server-1 kernel: [] SyS_ioctl+0xa1/0xc0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] system_call_fastpath+0x16/0x1b Feb 16 23:01:24 glusterfs-server-1 kernel: INFO: Object 0xffff880418bd5000 @offset=0 Feb 16 23:01:24 glusterfs-server-1 kernel: INFO: Object 0xffff880418bd5100 @offset=256 Feb 16 23:01:24 glusterfs-server-1 kernel: INFO: Object 0xffff880418bd5200 @offset=512 Feb 16 23:01:24 glusterfs-server-1 kernel: INFO: Object 0xffff880418bd5300 @offset=768 Feb 16 23:01:24 glusterfs-server-1 kernel: ============================================================================= Feb 16 23:01:24 glusterfs-server-1 kernel: BUG kmalloc-256(5:de25949f9c337499646227875f90b6e096531cb23e84bcac17802d355fd3cd05) (Tainted: G B ------------ ): Objects remaining in kmalloc-256(5:de25949f9c337499646227875f90b6e096531cb23e84bcac17802d355fd3cd05 Feb 16 23:01:24 glusterfs-server-1 kernel: ----------------------------------------------------------------------------- Feb 16 23:01:24 glusterfs-server-1 kernel: INFO: Slab 0xffffea00102af140 objects=16 used=1 fp=0xffff88040abc5f00 flags=0x2fffff00000080 Feb 16 23:01:24 glusterfs-server-1 kernel: CPU: 2 PID: 4911 Comm: lvcreate Tainted: G B ------------ 3.10.0-693.5.2.el7.x86_64 #1 Feb 16 23:01:24 glusterfs-server-1 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 Feb 16 23:01:24 glusterfs-server-1 kernel: ffffea00102af140 0000000069a8e6fa ffff8803ca3f7aa0 ffffffff816a3e51 Feb 16 23:01:24 glusterfs-server-1 kernel: ffff8803ca3f7b78 ffffffff811dbec4 0000000000000020 ffff8803ca3f7b88 Feb 16 23:01:24 glusterfs-server-1 kernel: ffff8803ca3f7b38 656a624f00000297 616d657220737463 6e6920676e696e69 Feb 16 23:01:24 glusterfs-server-1 kernel: Call Trace: Feb 16 23:01:24 glusterfs-server-1 kernel: [] dump_stack+0x19/0x1b Feb 16 23:01:24 glusterfs-server-1 kernel: [] slab_err+0xb4/0xe0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? kmalloc+0x1e3/0x230 Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? kmem_cache_close+0x127/0x2e0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] kmem_cache_close+0x149/0x2e0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] __kmem_cache_shutdown+0x14/0x80 Feb 16 23:01:24 glusterfs-server-1 kernel: [] kmem_cache_destroy+0x44/0xf0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] kmem_cache_destroy_memcg_children+0x89/0xb0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] kmem_cache_destroy+0x19/0xf0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] bioset_free+0xce/0x110 Feb 16 23:01:24 glusterfs-server-1 kernel: [] dm_destroy+0x1ae/0x310 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] dm_destroy+0x13/0x20 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] dev_remove+0x11e/0x180 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? dev_suspend+0x250/0x250 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] ctl_ioctl+0x1e5/0x500 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] dm_ctl_ioctl+0x13/0x20 [dm_mod] Feb 16 23:01:24 glusterfs-server-1 kernel: [] do_vfs_ioctl+0x33d/0x540 Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? file_has_perm+0x9f/0xb0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] ? ____fput+0xe/0x10 Feb 16 23:01:24 glusterfs-server-1 kernel: [] SyS_ioctl+0xa1/0xc0 Feb 16 23:01:24 glusterfs-server-1 kernel: [] system_call_fastpath+0x16/0x1b Feb 16 23:01:24 glusterfs-server-1 kernel: INFO: Object 0xffff88040abc5100 @offset=256

....

Feb 17 00:40:52 glusterfs-server-1 kernel: [] system_call_fastpath+0x16/0x1b Feb 17 00:40:52 glusterfs-server-1 kernel: INFO: Object 0xffff8800aa5b6300 @offset=768 Feb 17 00:40:52 glusterfs-server-1 kernel: INFO: Object 0xffff8800aa5b6900 @offset=2304 Feb 17 00:40:52 glusterfs-server-1 kernel: INFO: Object 0xffff8800aa5b6b00 @offset=2816 Feb 17 00:40:52 glusterfs-server-1 kernel: INFO: Object 0xffff8800aa5b6e00 @offset=3584 Feb 17 00:40:52 glusterfs-server-1 kernel: ============================================================================= Feb 17 00:40:52 glusterfs-server-1 kernel: BUG kmalloc-256(39:72308fee6abf4a5754e16aa78ef4e216cbf47f739366bd6aa4a4a9dc2e605119) (Tainted: G B ------------ ): Objects remaining in kmalloc-256(39:72308fee6abf4a5754e16aa78ef4e216cbf47f739366bd6aa4a4a9dc2e60511 Feb 17 00:40:52 glusterfs-server-1 kernel: ----------------------------------------------------------------------------- Feb 17 00:40:52 glusterfs-server-1 kernel: INFO: Slab 0xffffea000f299dc0 objects=16 used=12 fp=0xffff8803ca677200 flags=0x2fffff00000080 Feb 17 00:40:52 glusterfs-server-1 kernel: CPU: 3 PID: 21274 Comm: lvcreate Tainted: G B ------------ 3.10.0-693.5.2.el7.x86_64 #1 Feb 17 00:40:52 glusterfs-server-1 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 Feb 17 00:40:52 glusterfs-server-1 kernel: ffffea000f299dc0 0000000056649489 ffff8803bcb3f9c0 ffffffff816a3e51 Feb 17 00:40:52 glusterfs-server-1 kernel: ffff8803bcb3fa98 ffffffff811dbec4 ffff880300000020 ffff8803bcb3faa8 Feb 17 00:40:52 glusterfs-server-1 kernel: ffff8803bcb3fa58 656a624f81910018 616d657220737463 6e6920676e696e69 Feb 17 00:40:52 glusterfs-server-1 kernel: Call Trace: Feb 17 00:40:52 glusterfs-server-1 kernel: [] dump_stack+0x19/0x1b Feb 17 00:40:52 glusterfs-server-1 kernel: [] slab_err+0xb4/0xe0 `

jarrpa commented 6 years ago

@raghavendra-talur Does this ring any bells in Gluster-land?

max8899 commented 6 years ago

see this: https://github.com/moby/moby/issues/29879