cshivashankar
commented
4 years ago Ever since I moved to new kernel for 1.14 I am facing lot of issues in the nodes and nodes are going out quite frequently.Some times it doesn't become responsive and be in the ready status and sometimes it just flips to node not ready status. Is there any bug in Kernel which I am hitting into ? If there is any customization which is not relevant to the 1.14 , I can do the changes and test. Any suggestions would be greatly welcome
Following are the soft lock up messages seen
kernel: ena 0000:00:08.0 eth3: Found a Tx that wasn't completed on time, qid 4, index 787.
kernel: watchdog: BUG: soft lockup - CPU#58 stuck for 23s! [kworker/u146:7:707658]
kernel: Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xfrm_user xfrm_algo br_netfilter bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables veth iptable_mangle xt_connmark nf_conntrack_netlink nfnetlink xt_statistic xt_recent ipt_REJECT nf_reject_ipv4 xt_addrtype xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_mark iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter xt_conntrack nf_nat nf_conntrack overlay sunrpc crc32_pclmul ghash_clmulni_intel pcbc mousedev aesni_intel psmouse evdev aes_x86_64 crypto_simd glue_helper pcc_cpufreq button cryptd ena ip_tables x_tables xfs libcrc32c nvme crc32c_intel nvme_core ipv6 crc_ccitt autofs4
kernel: CPU: 58 PID: 707658 Comm: kworker/u146:7 Tainted: G L 4.14.165-133.209.amzn2.x86_64 #1
kernel: Hardware name: Amazon EC2 c5.18xlarge/, BIOS 1.0 10/16/2017
kernel: Workqueue: writeback wb_workfn (flush-259:0)
kernel: task: ffff8893fefa0000 task.stack: ffffc9002daec000
kernel: RIP: 0010:__list_del_entry_valid+0x28/0x90
kernel: RSP: 0018:ffffc9002daefcc0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
kernel: RAX: ffff88916d19b470 RBX: ffffc9002daefce8 RCX: dead000000000200
kernel: RDX: ffff88804b1f36c8 RSI: ffff888013237e08 RDI: ffff888013237e08
kernel: RBP: ffff88916d19b470 R08: ffff889488d1eb48 R09: 0000000180400037
kernel: R10: ffffc9002daefe10 R11: 0000000000000000 R12: ffff88916d608800
kernel: R13: ffff888013237e08 R14: ffffc9002daefd78 R15: ffff889488d1eb48
kernel: FS: 0000000000000000(0000) GS:ffff88a371380000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 000010923314f000 CR3: 000000000200a002 CR4: 00000000007606e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: PKRU: 55555554
kernel: Call Trace:
kernel: move_expired_inodes+0x6a/0x230
kernel: queue_io+0x61/0xf0
kernel: wb_writeback+0x258/0x300
kernel: ? wb_workfn+0xdf/0x370
kernel: ? __local_bh_enable_ip+0x6c/0x70
kernel: wb_workfn+0xdf/0x370
kernel: ? __switch_to_asm+0x41/0x70
kernel: ? __switch_to_asm+0x35/0x70
kernel: process_one_work+0x17b/0x380
kernel: worker_thread+0x2e/0x390
kernel: ? process_one_work+0x380/0x380
kernel: kthread+0x11a/0x130
kernel: ? kthread_create_on_node+0x70/0x70
kernel: ret_from_fork+0x35/0x40Ideally with the change https://github.com/awslabs/amazon-eks-ami/pull/367 , kubelet should be having plenty of resources and if there are high resource consuming pods , it should just evict it rather than node getting crashed.
I digged into the customizations in detail and following are the customizations done for the AMI used in environment:
Customizations :
-
Docker daemon related
1a. Docker env docker.env is used
# The max number of open files for the daemon itself, and all
# running containers. The default value of 1048576 mirrors the value
# used by the systemd service unit.
DAEMON_MAXFILES=1048576
# Additional startup options for the Docker daemon, for example:
# OPTIONS="--ip-forward=true --iptables=true"
# By default we limit the number of open files per container
# Options are moved to docker-daemon.json
#OPTIONS="--default-ulimit nofile=262144:1048576 --max-concurrent-downloads=100 --max-concurrent-uploads=100 --iptables=false"
# How many seconds the sysvinit script waits for the pidfile to appear
# when starting the daemon.
DAEMON_PIDFILE_TIMEOUT=101b. docker daemon json Default ulimits are changed and iptables is also set as false
{
"bridge": "none",
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "10"
},
"live-restore": true,
"iptables": false,
"max-concurrent-downloads": 100,
"max-concurrent-uploads": 100,
"default-ulimits": {
"nofile": {
"Name": "nofile",
"Hard": 1048576,
"Soft": 262144
}
}
}- System customizations:
2a. 20-nproc.conf Limit user process to avoid fork bombing
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
* soft nproc 3934939
root soft nproc unlimited2b. sysctl.conf
# Filesystem
# Maximum number of allowable concurrent requests
fs.aio-max-nr = 1048576
# 1M max_user_watches takes extra 1Gb of kernel memory per real UID
fs.inotify.max_user_watches = 1048576
fs.inotify.max_user_instances = 2048
# Kernel
# Prevents Docker from crashing with:
# > runtime/cgo: pthread_create failed: Resource temporarily unavailable
kernel.pid_max = 1048576
# Network
# Max number of packets queued when the interface receives packets faster than
# kernel can process them. Default is 1000
net.core.netdev_max_backlog = 5000
# Max rx/tx buffer sizes. Default 212k sounds too small for our 10g links, but
# setting some very high numbers may have nasty consequences too
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
# Threshold levels for ARP cache increased after neighbor table overflowed on DL1
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 8192
# Number of memory-mapped areas per process used by elasticsearch and other
# software that uses mmap
vm.max_map_count = 262144
~~2c. systemd.conf [Manager]
Cgroup controllers here are joined to make system.slice appear in the common
# cgroup heirarchy. This way kubelet is able to use "system.slice" as
# --system-reserved to isolate system resources from pod resources
JoinControllers=cpu,cpuacct,cpuset,net_cls,net_prio,hugetlb,memory~~-
Kubelet customizations
Following is the kubelet output with ps Eviction and system reserved option might be removed as its set in default.
/usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime docker --read-only-port=10255 --network-plugin cni --node-ip=10.77.94.123 --pod-infra-container-image=602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause-amd64:3.1 --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --kube-api-qps=30 --kube-api-burst=50 --event-qps=30 --event-burst=50 --registry-qps=0 --enforce-node-allocatable=pods --eviction-hard=imagefs.available<15%,memory.available<5%,nodefs.available<10%,nodefs.inodesFree<5% --serialize-image-pulls=false --v=3 --pod-max-pids=8192 --node-labels=node-role.kubernetes.io/spot-worker --resolv-conf=/etc/resolv.kubelet
/usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime docker --read-only-port=10255 --network-plugin cni --node-ip=10.77.94.123 --pod-infra-container-image=602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause-amd64:3.1 --eviction-hard=imagefs.available<15%,memory.available<5%,nodefs.available<10%,nodefs.inodesFree<5% --serialize-image-pulls=false --v=3 --pod-max-pids=8192 --resolv-conf=/etc/resolv.kubelet
resolv.kubelet is following
/usr/bin/grep -v "^search " /etc/resolv.conf > /etc/resolv.kubelet
EDIT: Edited this post after removing some customization
edubacco
colandre
eeeschwartz
jae-63
otterley
deniss-rostkovskis
JacobHenner
brettplarson
robertgates55
raravena80
mmerkes
SQUIDwarrior
Status Summary
This section added by @mmerkes
The AWS EKS team has a consistent repro and has engaged with the AmazonLinux team to root cause the issue and fix it. AmazonLinux has merged a patch that solves this issue in our repro and should work for customers, and that is now available via
yum update. Once you've updated your kernel and rebooted your instance, you should be runningkernel.x86_64 0:4.14.203-156.332.amzn2(or greater). All EKS optimized AMIs Packer versionv20201112or later will include this patch. Users have 2 options for fixing this issue:yum updateHere's the commands you need to patch your instances:
Original Issue
This original content from @cshivashankar
What happened: Node in the cluster becomes unresponsive and pods running on it also becomes unresponsive. As per the analysis and logs provided in AWS Case 6940959821, it was informed that this is observed when high IOPS is observed and a soft lock up happens which causes node to become unresponsive. Further investigation might be required .
What you expected to happen: Node should not crash or become unresponsive , if that was the case , control plane should identify it and mark it as not ready. State should be either node is ready and working properly or node is unresponsive and not ready and should be eventually removed from the cluster.
How to reproduce it (as minimally and precisely as possible): As per the analysis in the AWS case 6940959821 , the issue could be reproduced by having higher IOPS than the capacity of EBS for sustained amount of time.
Anything else we need to know?: This issue is being observed recently and I want to rule it out if it was due to using AMI of version 1.14 as we never observed this issue in 1.13. Is there any kernel bug that I am hitting into? For building the AMI, I cloned the "amazon/aws-eks-ami" repo and did the following changes ~~1. Installed Zabbix agent
Logs can be accessed in the AWS Case mentioned above Environment:
aws eks describe-cluster --name <name> --query cluster.platformVersion): "eks.9"aws eks describe-cluster --name <name> --query cluster.version): "1.14"uname -a): Linuxcat /etc/eks/releaseon a node):