User friendly network namespace names

SPYFF commented 3 weeks ago

Problem: Kubernetes and Mininet create anonymous network namespaces. These namespaces are invisible to Linux tools e.g.: ip netns list. They only have an ID (their inode number) which is not meaningful for the user. Namespaces created with ip netns add are added to /var/run/netns/. It's easy to interact with these namespaces e.g.: ip netns exec or delete them.

Goal: Create named namespace entries for Kubernetes pods. Tools can use this information to resolve pod names by namespace ID. For example pwru tool with that information can show the Pod name for the packets. Similarly for Mininet, the emulated host or router names can be presented by pwru.

Proposed solution (sketch):

List the available namespaces with lsns -t net command
In the output, we have namespace IDs and PIDs
Somehow we need to find the pod names the PIDs associated with
Manually create network namespace entry with the pod's name and the PID: ip netns attach NSNAME PID

Example:

lsns -t net

        NS TYPE NPROCS   PID USER       NETNSID NSFS                           COMMAND
...
4026533614 net       3 72512 65535   unassigned                                /pause
...

# we have anonymous netns with ID 4026533614 associated to PID 72512
# this is the PID of the process running in the pod named "tx"
# so we can create the netns manually with the following command

ip netns attach tx 72512

Missing pieces:

reliable way the find PIDs in a Pod (or other Kubernetes entity)
avoid namespace name collisions (Kubernetes namespaces might be transparent to Linux)
extend pwru to output the netns name instead of netns ID

levaitamas commented 3 weeks ago

just for the record: work is ongoing in the branch netnsid

levaitamas commented 3 weeks ago

what works:

2 namespaces: ns1 and ns2 they are connected via a veth pair:

┌─────────────────────────┐                   ┌─────────────────────────┐
│                         │                   │                         │
│                         │                   │                         │
│                 ┌───────┤                   ├───────┐                 │
│                 │ veth0 ╞═══════════════════╡ veth1 │                 │
│                 └───────┤                   ├───────┘                 │
│ns1                      │                   │                      ns2│
└─────────────────────────┘                   └─────────────────────────┘

demo, we ping one from the other (note the new NETNSID column):

SKB                CPU PROCESS          NETNS      NETNSID  MARK/x        IFACE       PROTO  MTU   LEN   TUPLE FUNC
[...]
0xffff97b232720e00 2   <empty>:365113   4026532330 ns1      0               12        0x0800 1500  84    192.168.64.1:0->192.168.64.2:0(icmp) ip_finish_output
0xffff97b232720e00 2   <empty>:365113   4026532330 ns1      0               12        0x0800 1500  84    192.168.64.1:0->192.168.64.2:0(icmp) __ip_finish_output
[...]
0xffff97b232720e00 2   <empty>:365113   4026534954 ns2      0               11        0x0800 1500  84    192.168.64.1:0->192.168.64.2:0(icmp) __netif_receive_skb
0xffff97b232720e00 2   <empty>:365113   4026534954 ns2      0               11        0x0800 1500  84    192.168.64.1:0->192.168.64.2:0(icmp) __netif_receive_skb_one_core
0xffff97b232720e00 2   <empty>:365113   4026534954 ns2      0               11        0x0800 1500  84    192.168.64.1:0->192.168.64.2:0(icmp) ip_rcv
[...]
0xffff97b232720e00 2   <empty>:365113   4026534954 ns2      0               11        0x0800 65536 64    192.168.64.1:0->192.168.64.2:0(icmp) icmp_rcv
0xffff97b232720e00 2   <empty>:365113   4026534954 ns2      0               11        0x0800 65536 64    192.168.64.1:0->192.168.64.2:0(icmp) __skb_checksum_complete
0xffff97b232720e00 2   <empty>:365113   4026534954 ns2      0               11        0x0800 65536 56    192.168.64.1:0->192.168.64.2:0(icmp) icmp_echo
0xffff97b232720e00 2   <empty>:365113   4026534954 ns2      0               11        0x0800 65536 56    192.168.64.1:0->192.168.64.2:0(icmp) icmp_reply
0xffff97b232720e00 2   <empty>:365113   4026534954 ns2      0               11        0x0800 65536 56    192.168.64.1:0->192.168.64.2:0(icmp) __ip_options_echo
[...]
0xffff97b232721000 2   <empty>:365113   4026534954 ns2      0               11        0x0800 1500  98    192.168.64.2:0->192.168.64.1:0(icmp) eth_type_trans
0xffff97b232721000 2   <empty>:365113   4026532330 ns1      0               12        0x0800 1500  84    192.168.64.2:0->192.168.64.1:0(icmp) __netif_rx
0xffff97b232721000 2   <empty>:365113   4026532330 ns1      0               12        0x0800 1500  84    192.168.64.2:0->192.168.64.1:0(icmp) netif_rx_internal
[...]
0xffff97b232721000 2   <empty>:365113   4026532330 ns1      0               12        0x0800 1500  84    192.168.64.2:0->192.168.64.1:0(icmp) __netif_receive_skb
0xffff97b232721000 2   <empty>:365113   4026532330 ns1      0               12        0x0800 1500  84    192.168.64.2:0->192.168.64.1:0(icmp) __netif_receive_skb_one_core
0xffff97b232721000 2   <empty>:365113   4026532330 ns1      0               12        0x0800 1500  84    192.168.64.2:0->192.168.64.1:0(icmp) ip_rcv
0xffff97b232721000 2   <empty>:365113   4026532330 ns1      0               12        0x0800 1500  84    192.168.64.2:0->192.168.64.1:0(icmp) ip_rcv_core
[...]
0xffff97b232721000 2   <empty>:365113   4026532330 ns1      0               12        0x0800 65536 64    192.168.64.2:0->192.168.64.1:0(icmp) icmp_rcv
0xffff97b232721000 2   <empty>:365113   4026532330 ns1      0               12        0x0800 65536 64    192.168.64.2:0->192.168.64.1:0(icmp) __skb_checksum_complete
0xffff97b232721000 2   <empty>:365113   4026532330 ns1      0               12        0x0800 65536 56    192.168.64.2:0->192.168.64.1:0(icmp) ping_rcv
[...]
0xffff97b232720e00 2   <empty>:365113   4026532330 ns1      0               0         0x0800 0     84    192.168.64.2:0->192.168.64.1:0(icmp) sock_rfree
0xffff97b232720e00 2   <empty>:365113   4026532330 ns1      0               0         0x0800 0     84    192.168.64.2:0->192.168.64.1:0(icmp) skb_release_data
0xffff97b232720e00 2   <empty>:365113   4026532330 ns1      0               0         0x0800 0     84    192.168.64.2:0->192.168.64.1:0(icmp) skb_free_head
0xffff97b232720e00 2   <empty>:365113   4026532330 ns1      0               0         0x0800 0     84    192.168.64.2:0->192.168.64.1:0(icmp) kfree_skbmem

future work:

test in mininet
add support for k8s pods

SPYFF commented 5 days ago

As described here, we can find out what is the PID or a pod's container: https://bpfman.io/v0.5.4/blog/2024/02/26/technical-challenges-for-attaching-ebpf-programs-in-containers/#finding-the-pid

a quick test with minikube:

kubectl run test --image=busybox -- sleep infinity
minikube ssh
docker inspect -f '{{.State.Pid}}' c38dcaf19fb7
3454

SPYFF / pwru

User friendly network namespace names #1