rfuchs
commented
3 months ago Try with a supported version, either 11.5 LTS or current 12.4 or master.
Open jmordica opened 3 months ago
rfuchs
commented
3 months ago Try with a supported version, either 11.5 LTS or current 12.4 or master.
jmordica
commented
3 months ago ok will try with 12.4 and report back.
jmordica
commented
3 months ago Tried with 12.4.1.7+0~mr12.4.1.7 git-HEAD-07244a2f
Same results.
It's very strange that this all of a sudden started happening with no change to the vm. Non-kernel mode works fine.
jmordica
commented
3 months ago Confirmed that the kernel module is running but this is what it shows when doing cat /proc/rtpengine/0/list
local inet4 10.128.15.216:13178
expect inet4 216.221.155.74:14552
src mismatch action: drop
stats: 104060 bytes, 605 packets, 0 errors
RTP payload type 0: 104060 bytes, 605 packets
last packet: 1722091146 SSRC in: 38686aaa [seq 605/0], e8d69f04 [seq 238/0]
options: RTP PT-filter SSRC-tracking forward-RTCP
output #0
src inet4 10.128.15.216:14726
dst inet4 10.12.4.2:17272
stats: 104060 bytes, 605 packets, 0 errors
SSRC out: 0 [seq 605+0/0], 0 [seq 238+0/0]
local inet4 10.128.15.216:14726
expect inet4 10.12.4.2:17272
src mismatch action: drop
stats: 29412 bytes, 171 packets, 0 errors
RTP payload type 0: 29412 bytes, 171 packets
last packet: 0 SSRC in: 7f91786 [seq 19193/0]
options: RTP PT-filter SSRC-tracking forward-RTCP
output #0
src inet4 10.128.15.216:13178
dst inet4 216.221.155.74:14552
stats: 29412 bytes, 171 packets, 0 errors
SSRC out: 0 [seq 0+0/0]
local inet4 10.128.15.216:14727
expect inet4 10.12.4.2:17273
src mismatch action: drop
stats: 180 bytes, 2 packets, 0 errors
last packet: 0 SSRC in: 7f91786 [seq 19193/0]
options: RTP RTCP SSRC-tracking forward-RTCP
output #0 (RTCP)
src inet4 10.128.15.216:13179
dst inet4 216.221.155.74:14553
stats: 180 bytes, 2 packets, 0 errors
SSRC out: 0 [seq 0+0/0]
local inet4 10.128.15.216:16752
expect inet4 104.3.145.160:4006
src mismatch action: drop
stats: 60888 bytes, 354 packets, 0 errors
RTP payload type 0: 60888 bytes, 354 packets
RTP payload type 101: 0 bytes, 0 packets
last packet: 1722091146 SSRC in: 48df170e [seq 29525/0]
options: RTP PT-filter SSRC-tracking forward-RTCP
output #0
src inet4 10.128.15.216:29332
dst inet4 10.12.4.2:10730
stats: 60888 bytes, 354 packets, 0 errors
SSRC out: 0 [seq 29525+0/0]
local inet4 10.128.15.216:16753
expect inet4 104.3.145.160:4007
src mismatch action: drop
stats: 280 bytes, 4 packets, 0 errors
last packet: 0 SSRC in: 48df170e [seq 29525/0]
options: RTP RTCP SSRC-tracking forward-RTCP
output #0 (RTCP)
src inet4 10.128.15.216:29333
dst inet4 10.12.4.2:10731
stats: 280 bytes, 4 packets, 0 errors
SSRC out: 0 [seq 29525+0/0]
local inet4 10.128.15.216:29332
expect inet4 10.12.4.2:10730
src mismatch action: drop
stats: 29756 bytes, 173 packets, 0 errors
RTP payload type 0: 29756 bytes, 173 packets
last packet: 0 SSRC in: 7c8a8fa2 [seq 10923/0]
options: RTP PT-filter SSRC-tracking forward-RTCP
output #0
src inet4 10.128.15.216:16752
dst inet4 104.3.145.160:4006
stats: 29756 bytes, 173 packets, 0 errors
SSRC out: 0 [seq 0+0/0]
local inet4 10.128.15.216:29333
expect inet4 10.12.4.2:10731
src mismatch action: drop
stats: 100 bytes, 1 packets, 0 errors
last packet: 0 SSRC in: 7c8a8fa2 [seq 10923/0]
options: RTP RTCP SSRC-tracking forward-RTCP
output #0 (RTCP)
src inet4 10.128.15.216:16753
dst inet4 104.3.145.160:4007
stats: 100 bytes, 1 packets, 0 errors
SSRC out: 0 [seq 0+0/0]Also confirmed nft list ruleset:
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain DOCKER {
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
XT match comment not found
counter packets 507 bytes 34625 jump CILIUM_POST_nat
XT match comment not found
counter packets 734 bytes 49186 jump KUBE-POSTROUTING
XT match comment not found
XT match addrtype not found
counter packets 657 bytes 44777 jump IP-MASQ
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
XT match comment not found
counter packets 17 bytes 1007 jump CILIUM_PRE_nat
iifname != "eth0" meta l4proto tcp ip daddr 169.254.169.254 XT match tcp not found
XT match comment not found
counter packets 0 bytes 0 XT target DNAT not found
iifname != "eth0" meta l4proto tcp ip daddr 169.254.169.254 XT match tcp not found
XT match comment not found
counter packets 0 bytes 0 XT target DNAT not found
}
chain OUTPUT {
type nat hook output priority -100; policy accept;
XT match comment not found
counter packets 507 bytes 34625 jump CILIUM_OUTPUT_nat
}
chain IP-MASQ {
ip daddr 169.254.0.0/16 XT match comment not found
counter packets 463 bytes 33137 return
ip daddr 10.0.0.0/8 XT match comment not found
counter packets 34 bytes 2040 return
ip daddr 172.16.0.0/12 XT match comment not found
counter packets 0 bytes 0 return
ip daddr 192.168.0.0/16 XT match comment not found
counter packets 0 bytes 0 return
ip daddr 240.0.0.0/4 XT match comment not found
counter packets 0 bytes 0 return
ip daddr 192.0.2.0/24 XT match comment not found
counter packets 0 bytes 0 return
ip daddr 198.51.100.0/24 XT match comment not found
counter packets 0 bytes 0 return
ip daddr 203.0.113.0/24 XT match comment not found
counter packets 0 bytes 0 return
ip daddr 100.64.0.0/10 XT match comment not found
counter packets 0 bytes 0 return
ip daddr 198.18.0.0/15 XT match comment not found
counter packets 0 bytes 0 return
ip daddr 192.0.0.0/24 XT match comment not found
counter packets 0 bytes 0 return
ip daddr 192.88.99.0/24 XT match comment not found
counter packets 0 bytes 0 return
XT match comment not found
counter packets 160 bytes 9600 XT target MASQUERADE not found
}
chain KUBE-MARK-DROP {
counter packets 0 bytes 0 XT target MARK not found
}
chain KUBE-MARK-MASQ {
counter packets 0 bytes 0 XT target MARK not found
}
chain KUBE-POSTROUTING {
XT match mark not found
counter packets 731 bytes 48858 return
counter packets 0 bytes 0 XT target MARK not found
XT match comment not found
counter packets 0 bytes 0 XT target MASQUERADE not found
}
chain KUBE-KUBELET-CANARY {
}
chain CILIUM_POST_nat {
}
chain CILIUM_OUTPUT_nat {
}
chain CILIUM_PRE_nat {
}
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain DOCKER {
}
chain DOCKER-ISOLATION-STAGE-1 {
iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
counter packets 0 bytes 0 return
}
chain DOCKER-ISOLATION-STAGE-2 {
oifname "docker0" counter packets 0 bytes 0 drop
counter packets 0 bytes 0 return
}
chain FORWARD {
type filter hook forward priority filter; policy drop;
XT match comment not found
counter packets 0 bytes 0 jump CILIUM_FORWARD
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-1
oifname "docker0" XT match conntrack not found
counter packets 0 bytes 0 accept
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
meta l4proto tcp counter packets 0 bytes 0 accept
meta l4proto udp counter packets 0 bytes 0 accept
meta l4proto icmp counter packets 0 bytes 0 accept
meta l4proto sctp counter packets 0 bytes 0 accept
}
chain DOCKER-USER {
counter packets 0 bytes 0 return
}
chain KUBE-FIREWALL {
ip saddr != 127.0.0.0/8 ip daddr 127.0.0.0/8 XT match comment not found
XT match conntrack not found
counter packets 0 bytes 0 drop
XT match comment not found
XT match mark not found
counter packets 0 bytes 0 drop
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
XT match comment not found
counter packets 38137 bytes 7488379 jump CILIUM_OUTPUT
counter packets 49755 bytes 8309024 jump KUBE-FIREWALL
}
chain INPUT {
type filter hook input priority filter; policy accept;
ip protocol udp counter packets 14 bytes 2318 jump rtpengine
XT match comment not found
counter packets 90180 bytes 929670982 jump CILIUM_INPUT
counter packets 125289 bytes 1267916661 jump KUBE-FIREWALL
}
chain KUBE-KUBELET-CANARY {
}
chain CILIUM_INPUT {
meta mark & 0x00000f00 == 0x00000200 XT match comment not found
counter packets 0 bytes 0 accept
}
chain CILIUM_OUTPUT {
meta mark & 0xfffffeff == 0x00000a00 XT match comment not found
counter packets 0 bytes 0 accept
meta mark & 0x00000e00 == 0x00000800 XT match comment not found
counter packets 0 bytes 0 accept
meta mark & 0x00000f00 != 0x00000e00 meta mark & 0x00000f00 != 0x00000d00 meta mark & 0x00000e00 != 0x00000a00 meta mark & 0x00000e00 != 0x00000800 meta mark & 0x00000f00 != 0x00000f00 XT match comment not found
counter packets 38137 bytes 7488379 XT target MARK not found
}
chain CILIUM_FORWARD {
oifname "cilium_host" XT match comment not found
counter packets 0 bytes 0 accept
iifname "cilium_host" XT match comment not found
counter packets 0 bytes 0 accept
iifname "lxc*" XT match comment not found
counter packets 0 bytes 0 accept
iifname "cilium_net" XT match comment not found
counter packets 0 bytes 0 accept
oifname "lxc*" XT match comment not found
counter packets 0 bytes 0 accept
iifname "lxc*" XT match comment not found
counter packets 0 bytes 0 accept
}
chain rtpengine {
XT target RTPENGINE not found
counter packets 14 bytes 2318
}
}
# Warning: table ip mangle is managed by iptables-nft, do not touch!
table ip mangle {
chain OUTPUT {
type route hook output priority mangle; policy accept;
meta l4proto tcp ip saddr 169.254.169.254 XT match tcp not found
counter packets 0 bytes 0 accept
meta l4proto udp ip saddr 169.254.169.254 XT match udp not found
counter packets 0 bytes 0 accept
ip saddr 169.254.169.254 counter packets 0 bytes 0 drop
}
chain KUBE-IPTABLES-HINT {
}
chain KUBE-KUBELET-CANARY {
}
chain CILIUM_POST_mangle {
}
chain CILIUM_PRE_mangle {
XT match socket not found
XT match comment not found
counter packets 0 bytes 0 XT target MARK not found
meta l4proto tcp meta mark 0x07940200 XT match comment not found
counter packets 0 bytes 0 XT target TPROXY not found
meta l4proto udp meta mark 0x07940200 XT match comment not found
counter packets 0 bytes 0 XT target TPROXY not found
}
chain POSTROUTING {
type filter hook postrouting priority mangle; policy accept;
XT match comment not found
counter packets 38137 bytes 7488379 jump CILIUM_POST_mangle
}
chain PREROUTING {
type filter hook prerouting priority mangle; policy accept;
XT match comment not found
counter packets 90180 bytes 929670982 jump CILIUM_PRE_mangle
}
}
table ip6 mangle {
chain KUBE-IPTABLES-HINT {
}
chain KUBE-KUBELET-CANARY {
}
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
chain KUBE-MARK-DROP {
counter packets 0 bytes 0 XT target MARK not found
}
chain KUBE-MARK-MASQ {
counter packets 0 bytes 0 XT target MARK not found
}
chain KUBE-POSTROUTING {
XT match mark not found
counter packets 3 bytes 240 return
counter packets 0 bytes 0 XT target MARK not found
XT match comment not found
counter packets 0 bytes 0 XT target MASQUERADE not found
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
XT match comment not found
counter packets 3 bytes 240 jump KUBE-POSTROUTING
}
chain KUBE-KUBELET-CANARY {
}
}
# Warning: table ip6 filter is managed by iptables-nft, do not touch!
table ip6 filter {
chain KUBE-FIREWALL {
XT match comment not found
XT match mark not found
counter packets 0 bytes 0 drop
}
chain KUBE-KUBELET-CANARY {
}
chain rtpengine {
XT target RTPENGINE not found
counter packets 0 bytes 0
}
chain INPUT {
type filter hook input priority filter; policy accept;
ip6 nexthdr udp counter packets 0 bytes 0 jump rtpengine
}
}
# Warning: table ip raw is managed by iptables-nft, do not touch!
table ip raw {
chain CILIUM_OUTPUT_raw {
oifname "lxc*" meta mark & 0xfffffeff == 0x00000a00 XT match comment not found
counter packets 0 bytes 0 XT target CT not found
oifname "cilium_host" meta mark & 0xfffffeff == 0x00000a00 XT match comment not found
counter packets 0 bytes 0 XT target CT not found
oifname "lxc*" meta mark & 0x00000e00 == 0x00000800 XT match comment not found
counter packets 0 bytes 0 XT target CT not found
oifname "cilium_host" meta mark & 0x00000e00 == 0x00000800 XT match comment not found
counter packets 0 bytes 0 XT target CT not found
}
chain CILIUM_PRE_raw {
meta mark & 0x00000f00 == 0x00000200 XT match comment not found
counter packets 0 bytes 0 XT target CT not found
}
chain OUTPUT {
type filter hook output priority raw; policy accept;
XT match comment not found
counter packets 38137 bytes 7488379 jump CILIUM_OUTPUT_raw
}
chain PREROUTING {
type filter hook prerouting priority raw; policy accept;
XT match comment not found
counter packets 90180 bytes 929670982 jump CILIUM_PRE_raw
}
}Still no dice.
jmordica
commented
3 months ago One notable observation: The environment is GKE and has been running for years. When the kubernetes master node got upgraded from 1.27 to 1.28 the rtpengine worker node immediately stopped working in kernel mode. The worker node itself didn't update to v1.28. It was still on the previous version with no restarts when the issue occurred.
The worker node has been moved to 1.28 and the issue still happens when the media is sent to the kernel. Have tried both 11.5 and 12.4.
rfuchs
commented
3 months ago Might be something specific to your VM setup then, and/or some interaction with other nft rules.
jmordica
commented
3 months ago Right. Not seeing any conflicts in the ruleset above?
rtpengine version the issue has been seen with
11.2.1.4+0~mr11.2.1.4 git-HEAD-1d93d9b4
Used distribution and its version
Ubuntu 22.04.3 LTS
Linux kernel version used
5.15.0-1048-gke
CPU architecture issue was seen on (see
uname -m)x86_64
Expected behaviour you didn't see
Audio continuing after entering kernel mode
Unexpected behaviour you saw
When kernelizing media stream happens (3-5 seconds after the call starts) one-way audio occurs.
Steps to reproduce the problem
Can reproduce on every call.
Additional program output to the terminal or logs illustrating the issue
Anything else?
rtpengine.conf:
This call is controlled by kamailio.