AWSNB
commented
4 years ago A bug was fixed in ENA XDP implementation, and release to github , release note quotes “* Fix XDP PASS issue due to incorrect handling of offset in rx_info.”
https://github.com/amzn/amzn-drivers/commit/ccbb1fe2c2f2ab3fc6d7827b012ba8ec06f32c39
could you try ena driver 2.2.8 from github to confirm if it solves the problem ? we working on release the fix to kernel (its probably there already, but not backported to LTS)
From: Augustinas Stirbis notifications@github.com Reply-To: amzn/amzn-drivers reply@reply.github.com Date: Thursday, May 28, 2020 at 12:22 AM To: amzn/amzn-drivers amzn-drivers@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [amzn/amzn-drivers] ~210 byte limit with eBPF (Cilium) on Linux Kernels 5.3/4* (#131)
When using EC2 type instance with ENA support (let's say T3a.medium) inbound & outbound traffic get's dropped if above ~210 bytes when using eBPF functionality in Cilium Kubernetes CNI (specifically kube-proxy-replacement) & using Linux Kernel 5.3 or 5.4.
To reproduce the issue Amazon Linux 2 AMI 2.0.20200406.0 x86_64 HVM gp2) or Ubuntu 19.10/20.04
cat <
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubectl, conntrack
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
&& chmod +x minikube
sudo cp minikube /usr/local/bin && rm minikube
echo "bpffs /sys/fs/bpf bpf defaults 0 0" >> /etc/fstab
amazon-linux-extras install docker
minikube start --network-plugin=cni --driver=none
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.7/install/kubernetes/quick-install.yaml
One can ping this VM from outside or ping from inside to internet ping -s 300 amazon.com
it should work, but after upgrading Linux Kernel to 5.4 sudo amazon-linux-extras install kernel-ng
After reboot when Cilium POD wakes up
[root@ip-172-31-44-20 ~]# ping -s 214 amazon.com
PING amazon.com (176.32.103.205) 214(242) bytes of data.
222 bytes from 176.32.103.205 (176.32.103.205): icmp_seq=1 ttl=227 time=85.6 ms
222 bytes from 176.32.103.205 (176.32.103.205): icmp_seq=2 ttl=227 time=85.6 ms
222 bytes from 176.32.103.205 (176.32.103.205): icmp_seq=3 ttl=227 time=85.6 ms
222 bytes from 176.32.103.205 (176.32.103.205): icmp_seq=4 ttl=227 time=85.6 ms
222 bytes from 176.32.103.205 (176.32.103.205): icmp_seq=5 ttl=227 time=85.6 ms
^C
--- amazon.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 85.620/85.632/85.646/0.185 ms
[root@ip-172-31-44-20 ~]# ping -s 215 amazon.com
PING amazon.com (176.32.98.166) 215(243) bytes of data.
^C
--- amazon.com ping statistics ---
46 packets transmitted, 0 received, 100% packet loss, time 46048ms
Powering off Instance and changing Instance family to T2 or M4 does not have the problem as these are ENA not supporting instances, switching back to T3 will have the issue
Kernel 4.19 worked fine, upgrade to 5.4 broke it
Disabing Cilium Kube-proxy-replacement=disabled also removes the problem
Hence ENA driver does not go well with specific eBPF magic on latest LTS kernels
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/amzn/amzn-drivers/issues/131, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFTRWCOJMMAC2GFPL7KYKMDRTYGFZANCNFSM4NM2GAIQ.
ShayAgros
AugustinasS
When using EC2 type instance with ENA support (let's say T3a.medium) inbound & outbound traffic get's dropped if above ~214 bytes when using eBPF functionality in Cilium Kubernetes CNI (specifically kube-proxy-replacement) & using Linux Kernel 5.3 or 5.4.
To reproduce the issue Amazon Linux 2 AMI 2.0.20200406.0 x86_64 HVM gp2) or Ubuntu 19.10/20.04
One can ping this VM from outside or ping from inside to internet
ping -s 300 amazon.comit should work, but after upgrading Linux Kernel to 5.4
sudo amazon-linux-extras install kernel-ngAfter reboot when Cilium POD wakes up
Powering off Instance and changing Instance family to T2 or M4 does not have the problem as these are ENA not supporting instances, switching back to T3 will have the issue
Kernel 4.19 worked fine, upgrade to 5.4 broke it
Disabing Cilium
Kube-proxy-replacement=disabledalso removes the problemHence ENA driver does not go well with specific eBPF magic on latest LTS kernels