search

k8snetworkplumbingwg / multus-cni

A CNI meta-plugin for multi-homed pods in Kubernetes
Apache License 2.0
2.43k stars 584 forks source link

PTP support when using CRD #30

Closed egernst closed 7 years ago

egernst commented 7 years ago

Steps:

  1. Copy multus configration file to /etc/cni/net.d/:

    {
"type": "multus",
"log_level": "debug",
"kubeconfig": "/etc/kubernetes/admin.conf",
"delegates": [{
    "name": "flannel",
    "type": "flannel",
    "masterplugin": true
} ]
}

  2. Start a single node kubernetes cluster: sudo -E kubeadm init --pod-network-cidr 10.244.0.0/16

  3. Startup flannel, using .yamls from the upstream coreOS repo:

    sudo -E kubectl create -f "../k8s/kube-flannel-rbac.yml"
sudo -E kubectl create --namespace kube-system -f "../k8s/kube-flannel.yml"

  4. Taint master so we can schedule work on it and create a custom resource definition for networks:

    master=$(hostname)
sudo -E kubectl taint nodes "$master" node-role.kubernetes.io/master:NoSchedule-
kubectl create -f crd-network.yaml

    Where crd-network yaml is:

    apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: networks.kubernetes.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: kubernetes.com
# version name to use for REST API: /apis/<group>/<version>
version: v1
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: networks
# singular name to be used as an alias on the CLI and for display
singular: network
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: Network
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- net

  5. Create a few CRD network types on the system, based on following yaml:

    
---
apiVersion: "kubernetes.com/v1"
kind: Network
metadata:
name: ptp-net
plugin: ptp
args: '[
{
    "name": "ptp-net",
    "type": "ptp",
    "ipam": {
              "type": "host-local",
              "subnet": "10.248.246.144/28",
              "routes": [
               { "dst": "0.0.0.0/0" }
    }
}
]'

apiVersion: "kubernetes.com/v1" kind: Network metadata: name: br-net-1 plugin: bridge args: '[ { "name": "br-net-1", "type": "bridge", "bridge": "br-net-1", "ipam": { "type": "host-local", "subnet": "10.1.10.0/24" } } ]'

apiVersion: "kubernetes.com/v1" kind: Network metadata: name: br-net-2 plugin: bridge args: '[ { "name": "br-net-2", "type": "bridge", "bridge": "br-net-2", "ipam": { "type": "host-local", "subnet": "11.1.1.0/24" } } ]'


6.  Start a pod making use of each of these three networks:

$ sudo -E kubectl create -f ptp-pod.yaml $ cat ptp-pod.yaml

apiVersion: v1 kind: Pod # TODO make these deployments later metadata: name: ptp-test annotations: networks: '[ { "name": "br-net-1" }, { "name": "br-net-2" }, { "name": "ptp-net" } ]' spec: containers:

  1. See following networks:
$ sudo -E kubectl exec -it ptp-test  -- bash
root@ptp-test:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if135: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 0a:58:0a:f4:00:44 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.68/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5433:ff:fe5f:9145/64 scope link 
       valid_lft forever preferred_lft forever
  1. Bring up one more pod, this time not using PTP:
sudo -E kubectl create -f test-pod.yaml 
 cat test-pod.yaml 
---
apiVersion: v1
kind: Pod # TODO make these deployments later
metadata:
  name: no-ptp-test
  annotations:
    networks: '[
        { "name": "br-net-1" },
        { "name": "br-net-2" }
    ]'
spec:
  containers:
  - name: no-ptp-test
    image: "ubuntu:14.04"
    stdin: true
    tty: true
  1. Network comes up without issue: 
    $ sudo -E kubectl exec -it no-ptp-test2 -- bash -c "ip a"
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host 
   valid_lft forever preferred_lft forever
3: eth0@if146: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
link/ether 0a:58:0a:01:0a:03 brd ff:ff:ff:ff:ff:ff
inet 10.1.10.3/24 scope global eth0
   valid_lft forever preferred_lft forever
inet6 fe80::8c3e:a1ff:fed1:8659/64 scope link 
   valid_lft forever preferred_lft forever
5: net0@if147: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
link/ether 0a:58:0b:01:01:03 brd ff:ff:ff:ff:ff:ff
inet 11.1.1.3/24 scope global net0
   valid_lft forever preferred_lft forever
inet6 fe80::2098:afff:febf:fbb0/64 scope link 
   valid_lft forever preferred_lft forever
egernst commented 7 years ago

Think I see the issue!

rkamudhan commented 7 years ago

@egernst if you attending the Intel's NFV features in Kubernetes Kubecon session , we doing hands-on session with portable Virutal box, and we giving out Intel USB stick with container exp kit as collateral. The collateral has all ansible script to be deployed.

egernst commented 7 years ago

Sorry, thought it was an issue with the ptp network yaml description, but looks like it should be okay from my perspective. I'm curious if you have tried this out? My goal is to be able to post some updated documentation / simple example configurations which can be curl'd for folks getting started with multus.

egernst commented 7 years ago

@rkamudhan -- I wish I was heading down - won't make it.

If you have ansible available now on a gist, etc, that's fine -- please share! I'd prefer a minimal number of steps and simple examples which are hw agnostic for getting up and running with multus. I was able to get it up with bridge and SRIOV (details @ https://github.com/egernst/k8s-testing-scripts/tree/master/multus-sriov), but am now having issue with PTP on my end.

I saw PTP using the conf in your README.md, but no examples of in CRD; is this something you've run on your end?

rkamudhan commented 7 years ago

Typo error man.

apiVersion: "kubernetes.com/v1"
kind: Network
metadata:
  name: ptp-net
plugin: ptp
args: '[
    {
        "name": "ptp-net",
        "type": "ptp",
        "ipam": {
                  "type": "host-local",
                  "subnet": "10.248.246.144/28",
                  "routes": [
                   { "dst": "0.0.0.0/0" }
        }
    }
]'

should be

apiVersion: "kubernetes.com/v1"
kind: Network
metadata:
  name: ptp-net
plugin: ptp
args: '[
    {
        "name": "ptp-net",
        "type": "ptp",
        "ipam": {
                  "type": "host-local",
                  "subnet": "10.248.246.144/28",
                  "routes": [
                   { "dst": "0.0.0.0/0" }
             ]
        }
    }
]'

Multus had error recovery mechanism using default network. Can you send me your conf file ? Whether it having default network? Some user wants this mechanism, for some, it is an issue.

egernst commented 7 years ago

Yep -- that's why I ended up closing the book 30 minutes ago. As soon as I typed it out I noticed the ].

Lame question -- where are the logs sent for this? I'll push the files onto a Quickstart.gist later tonight -- may be a good addition to have on the multus-cni site.

rkamudhan commented 7 years ago

Please let me know the PR, I like to include your testing script repo link in the Multus README.md

rkamudhan commented 7 years ago

@egernst Is the network in the pod annotation fails in the creation, then the network from default is picked by Multus ?

egernst commented 7 years ago

That was indeed the case. I ended up seeing it pull up my default from flannel. Thanks for the help!