Unable to make Traefik work

[Jean-Baptiste-Lasselle]

Hi, First thank you for sharing your work.

• Basically, your repo is one of the most serious I found with a recipe to deploy traefik as ingress controller

• I have tried a lot of different quick recipes, and I am having a very hard time finding just one configuration that would ever work

• I would so be grateful if you could tell me what I did wrong, using your recipe :

• I git clone your recipe, and in the root folder of your recipe, I executed exactly this :

# --- #
kubectl cluster-info
# --- #
kubectl create namespace traefik
# --- #
kubectl apply -f traefik/traefik-rbac.yaml
# --- #
kubectl apply -f traefik/traefik-definition.yaml
# --- #
kubectl apply -f traefik/traefik-deployment.yaml
# --- # Ici est défnit le nom de domaine pour l'accès à traefik
kubectl apply -f traefik/traefik-ingress-dashboard.yaml

echo ''
echo "Now you can access Traefik using Kubernetes credentials, and executing : "
echo ''
echo "  kubectl port-forward service/traefik-dashboard <ANY_PORT_NO_YOU_WANT>:8080 -n traefik "
echo ''
echo "after that, you will access Traefik 's web ui at http://127.0.0.1:<ANY_PORT_NO_YOU_WANT>'"
echo ''

kubectl create namespace cheese

kubectl apply -f cheese

• my output has no error, and after that, I can access Traefik v2 using kubectl port-forward
• But When I deploy the cheese App, well traefik detects them, I can see them appear into the traefik dashboard, and traefik tells me he manages the routes :

• But impossible to access any of the external urls :( :

jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ kubectl cluster-info
Kubernetes master is running at https://minikube.pegasusio.io:8443
KubeDNS is running at https://minikube.pegasusio.io:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ ping -c 4 minikube.pegasusio.io
PING minikube.pegasusio.io (192.168.1.21) 56(84) bytes of data.
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=1 ttl=64 time=0.249 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=2 ttl=64 time=0.301 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=3 ttl=64 time=0.228 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=4 ttl=64 time=0.249 ms

--- minikube.pegasusio.io ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3061ms
rtt min/avg/max/mdev = 0.228/0.256/0.301/0.033 ms
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ ping -c 4 cheeses.pegasusio.io
PING cheeses.pegasusio.io (192.168.1.21) 56(84) bytes of data.
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=1 ttl=64 time=0.303 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=2 ttl=64 time=0.307 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=3 ttl=64 time=0.270 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=4 ttl=64 time=0.269 ms

--- cheeses.pegasusio.io ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3078ms
rtt min/avg/max/mdev = 0.269/0.287/0.307/0.021 ms
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k https://minikube.pegasusio.io:8443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k https://minikube.pegasusio.io
curl: (7) Failed to connect to minikube.pegasusio.io port 443: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k http://minikube.pegasusio.io
curl: (7) Failed to connect to minikube.pegasusio.io port 80: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k http://traefik.pegasusio.io
curl: (7) Failed to connect to traefik.pegasusio.io port 80: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k https://traefik.pegasusio.io
curl: (7) Failed to connect to traefik.pegasusio.io port 443: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k https://cheeses.pegasusio.io
curl: (7) Failed to connect to cheeses.pegasusio.io port 443: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k https://cheeses.pegasusio.io/stilton
curl: (7) Failed to connect to cheeses.pegasusio.io port 443: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k https://cheeses.pegasusio.io/cheddar
curl: (7) Failed to connect to cheeses.pegasusio.io port 443: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k http://cheeses.pegasusio.io/cheddar
curl: (7) Failed to connect to cheeses.pegasusio.io port 80: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl -k http://cheeses.pegasusio.io/stilton
curl: (7) Failed to connect to cheeses.pegasusio.io port 80: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ kubectl get all -n cheese
NAME                               READY   STATUS    RESTARTS   AGE
pod/cheddar-5799cfc549-fdk4t       1/1     Running   0          10m
pod/cheddar-5799cfc549-vrfrt       1/1     Running   0          10m
pod/stilton-6cd9d85649-pnclc       1/1     Running   0          10m
pod/stilton-6cd9d85649-sz8pb       1/1     Running   0          10m
pod/wensleydale-68f848bf5c-5t5gt   1/1     Running   0          10m
pod/wensleydale-68f848bf5c-zh4z9   1/1     Running   0          10m

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/cheddar       ClusterIP   10.101.58.108   <none>        80/TCP    10m
service/stilton       ClusterIP   10.108.89.219   <none>        80/TCP    10m
service/wensleydale   ClusterIP   10.104.82.142   <none>        80/TCP    10m

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cheddar       2/2     2            2           10m
deployment.apps/stilton       2/2     2            2           10m
deployment.apps/wensleydale   2/2     2            2           10m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/cheddar-5799cfc549       2         2         2       10m
replicaset.apps/stilton-6cd9d85649       2         2         2       10m
replicaset.apps/wensleydale-68f848bf5c   2         2         2       10m
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ kubectl get logs -f service/cheddar -n cheese
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ kubectl logs -f service/cheddar -n cheese
Found 2 pods, using pod/cheddar-5799cfc549-fdk4t
^C
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ kubectl logs -f service/wensleydale -n cheese
Found 2 pods, using pod/wensleydale-68f848bf5c-5t5gt

• here is the output of commands which will give you infos (note I just changed domain name to match my VM ) :

jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ ping -c 4 http://traefik.pegasusio.io/
ping: http://traefik.pegasusio.io/: Name or service not known
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ ping -c 4 traefik.pegasusio.io
PING traefik.pegasusio.io (192.168.1.21) 56(84) bytes of data.
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=1 ttl=64 time=0.246 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=2 ttl=64 time=0.246 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=3 ttl=64 time=0.330 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=4 ttl=64 time=0.260 ms

--- traefik.pegasusio.io ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3058ms
rtt min/avg/max/mdev = 0.246/0.270/0.330/0.038 ms
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ ^C
jibl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl http://traefik.pegasusio.io/
curl: (7) Failed to connect to traefik.pegasusio.io port 80: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl https://traefik.pegasusio.io/
curl: (7) Failed to connect to traefik.pegasusio.io port 443: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ ping -c 4 ceeses.pegasusio.io
ping: ceeses.pegasusio.io: Name or service not known
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ ping -c 4 cheeses.pegasusio.io
PING cheeses.pegasusio.io (192.168.1.21) 56(84) bytes of data.
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=1 ttl=64 time=0.258 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=2 ttl=64 time=0.344 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=3 ttl=64 time=0.246 ms
64 bytes from dockhost1.marguerite.io (192.168.1.21): icmp_seq=4 ttl=64 time=0.279 ms

--- cheeses.pegasusio.io ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3051ms
rtt min/avg/max/mdev = 0.246/0.281/0.344/0.043 ms
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ ^C
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl http://cheeses.pegasusio.io
curl: (7) Failed to connect to cheeses.pegasusio.io port 80: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ 
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ curl http://cheeses.pegasusio.io/stilton
curl: (7) Failed to connect to cheeses.pegasusio.io port 80: Connection refused
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ cat traefik/traefik-ingress-dashboard.yaml 
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard-ingress
  namespace: traefik
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`traefik.pegasusio.io`)
    kind: Rule
    services:
    - name: traefik-dashboard
      port: 8080
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ cat cheese/cheese-ingress.yaml 
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: cheesestripprefix
  namespace: cheese
spec:
  stripPrefix:
    prefixes:
      - /stilton
      - /cheddar
      - /wensleydale

---

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: cheeses
  namespace: cheese
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`cheeses.pegasusio.io`) && PathPrefix(`/stilton`)
    kind: Rule
    services:
    - name: stilton
      port: 80
    middlewares:
      - name: cheesestripprefix
  - match: Host(`cheeses.pegasusio.io`) && PathPrefix(`/cheddar`)
    kind: Rule
    services:
    - name: cheddar
      port: 80
    middlewares:
      - name: cheesestripprefix
  - match: Host(`cheeses.pegasusio.io`) && PathPrefix(`/wensleydale`)
    kind: Rule
    services:
    - name: wensleydale
      port: 80
    middlewares:
      - name: cheesestripprefix
jbl@poste-devops-jbl-16gbram:~/atelier-traefik/essai1$ 

[Thoorium]

You mention that you use the kubectl proxy to access the traefik dashboard. The dashboard deployment should have assigned an IP to your host, which is traefik.pegasusio.io via Metallb and DHCP. Are you editing your host file to assign the IPs or using something else to make sure the DNS queries resolves?

Edit: I see the ping replies but I wonder why you use the proxy to access traefik :) Edit2: Everything resolves to the same IP, 192.168.1.21 which is wrong. If you look at the Kubernetes dashboard, you should be able to find the IPs metallb assigned to your applications.

[Jean-Baptiste-Lasselle]

oh hi @Thoorium I just finished completing writing the issue, I read and answer you now

[Jean-Baptiste-Lasselle]

@Thoorium

• I have one VM, and a minikube inside of it,
• I believe my minikube works fine, cause I ssuccessfully deployed the histroical traefik 1.7 example on branch v1.7 I 'm sure you heard of.
• So I just have a minkube, nothing special about it
• As you can see in the output I gived above, I can hit minikube Kubernetes API trhough 8443 no prob
• I did not use anything in your repo, except the traefik deployement steps
• So thank you for ansering, I m really, really having a hard time with that at work

[Jean-Baptiste-Lasselle]

oh, and yes, absolutely, I edited my /etc/hosts (note the 12.168.1.21 same address you see when I try ping cheeses.pegasusio.io) :

$ cat /etc/hosts|grep pegasusio.io
192.168.1.21    dashboard.pegasusio.io pegasusio.io
192.168.1.21    oci-registry.pegasusio.io pegasusio.io 
192.168.1.21    portus.pegasusio.io pegasusio.io
192.168.1.21    notary.pegasusio.io pegasusio.io
192.168.1.21    pegasusio.io 
192.168.1.21    minikube.pegasusio.io pegasusio.io
192.168.1.21 minikube.pegasusio.io
192.168.1.21 traefik.pegasusio.io 
192.168.1.21 gravitee-am.pegasusio.io
192.168.1.21 gravitee-apim.pegasusio.io
192.168.1.21 gravitee-am-ui.pegasusio.io
192.168.1.21 gravitee-apim-ui.pegasusio.io
192.168.1.21 stilton.pegasusio.io
192.168.1.21 cheddar.pegasusio.io
192.168.1.21 wensleydale.pegasusio.io
192.168.1.21 cheeses.pegasusio.io
192.168.1.21 minikube.pegasusio.io

[Thoorium]

Oh I see now. My setup relies on metallb assigning IPs to the deployments (IE: traefik) which is kinda the opposite of what you're doing, since you're working with only the host's IP. I can't test this right now but I think that modifying the traefik-deployment.yaml service deployment type from LoadBalancer to ClusterIp should expose traefik via the host's IP.

kind: Service
apiVersion: v1
metadata:
  name: traefik
  namespace: traefik
  annotations: {}
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - protocol: TCP
    port: 80
    name: http
  #- protocol: TCP
  #  port: 443
  #  name: https
  type: ClusterIp # <-- Here

Modify the file, apply the deployment and look in the dashboard if the assigned IP is the one from your host.

[Jean-Baptiste-Lasselle]

@Thoorium oh my gosh so thak you for your help, I'll try that, and tell you tomorrow, than k you so much again ! ttyt

[Jean-Baptiste-Lasselle]

@Thoorium Note : I will gladly take your advice on what I shoudl read to get a lot more stronger on kubernetes networking, when finished with this issue

[Jean-Baptiste-Lasselle]

Ok, I just tried before sleep, With all three possibilities:

---
kind: Service
apiVersion: v1
metadata:
  name: traefik
  namespace: traefik
  annotations: {}
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - protocol: TCP
    port: 80
    name: http
  #- protocol: TCP
  #  port: 443
  #  name: https
  # type: LoadBalancer
  # type: ClusterIP
  type: NodePort

• Results were :
  • with NodePort : Ok both traefik ingress and dashboard are accessible trough a highly numbered port no
  • with ClusterIP : I was puzzled, but no, that is just about only internal access in the Kubernetes network
  • with LoadBalancer : that is what I want, to get my traefik ingress on 80, and the dashboard, accessible trough this ingress. There I saw that I have a problem, I should get an ExternalIP, but it just remains in pending state forever :

$ kubectl  get service/traefik -n traefik -w
NAME      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
traefik   LoadBalancer   10.108.216.198   <pending>     80:31339/TCP   57s

• HAve you experienced similar problems with your work ?

• I event did a test :
  • on my one and only VM, that I can ping at multple hostnames like pegasusio.io, I ran this to launch a web app :

jbl@pegasusio:~$ git clone https://github.com/scotty-c/docker-demo-webapp
Clonage dans 'docker-demo-webapp'...
remote: Enumerating objects: 21, done.
remote: Total 21 (delta 0), reused 0 (delta 0), pack-reused 21
Dépaquetage des objets: 100% (21/21), fait.
jbl@pegasusio:~$ cd docker-demo-webapp/
jbl@pegasusio:~/docker-demo-webapp$ docker build -t scottyc/webapp .
Sending build context to Docker daemon  82.94kB
Step 1/11 : FROM golang:1.11.2-alpine3.8 as build
1.11.2-alpine3.8: Pulling from library/golang
4fe2ade4980c: Pull complete 
2e793f0ebe8a: Pull complete 
77995fba1918: Pull complete 
cacfaec3bb6b: Pull complete 
885a921d7cd2: Pull complete 
Digest: sha256:692eff58ac23cafc7cb099793feb00406146d187cd3ba0226809317952a9cf37
Status: Downloaded newer image for golang:1.11.2-alpine3.8
 ---> 57915f96905a
Step 2/11 : WORKDIR /go/src/github.com/scottyc/webapp
 ---> Running in 3443eb606f3d
Removing intermediate container 3443eb606f3d
 ---> f6e3e8718059
Step 3/11 : COPY web.go web.go
 ---> b6b57b30fae2
Step 4/11 : RUN CGO_ENABLED=0 GOOS=linux go build -o ./bin/webapp github.com/scottyc/webapp
 ---> Running in bdd0b27495aa
Removing intermediate container bdd0b27495aa
 ---> 2fc1b3a15be6
Step 5/11 : FROM alpine:3.8
3.8: Pulling from library/alpine
486039affc0a: Pull complete 
Digest: sha256:2bb501e6173d9d006e56de5bce2720eb06396803300fe1687b58a7ff32bf4c14
Status: Downloaded newer image for alpine:3.8
 ---> c8bccc0af957
Step 6/11 : RUN apk add --update vim &&     rm -rf /var/cache/apk/* &&     mkdir -p /web/static/
 ---> Running in 2ec64683445a
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/5) Installing lua5.3-libs (5.3.5-r2)
(2/5) Installing ncurses-terminfo-base (6.1_p20180818-r1)
(3/5) Installing ncurses-terminfo (6.1_p20180818-r1)
(4/5) Installing ncurses-libs (6.1_p20180818-r1)
(5/5) Installing vim (8.1.1365-r0)
Executing busybox-1.28.4-r3.trigger
OK: 39 MiB in 18 packages
Removing intermediate container 2ec64683445a
 ---> 7ef7f0decf59
Step 7/11 : COPY --from=build /go/src/github.com/scottyc/webapp/bin/webapp /usr/bin
 ---> bb3acc66b2d1
Step 8/11 : COPY index.html /web/static/index.html
 ---> 55ea98882e8d
Step 9/11 : WORKDIR /web
 ---> Running in 862ff572c41e
Removing intermediate container 862ff572c41e
 ---> 8ddb19e0f0bb
Step 10/11 : EXPOSE 3000
 ---> Running in b1b5e1493133
Removing intermediate container b1b5e1493133
 ---> 702f45123981
Step 11/11 : ENTRYPOINT webapp
 ---> Running in f980cbfba6c7
Removing intermediate container f980cbfba6c7
 ---> 4ab677b3d78c
Successfully built 4ab677b3d78c
Successfully tagged scottyc/webapp:latest
jbl@pegasusio:~/docker-demo-webapp$ docker run -d --name webapp -p 3000:3000 scottyc/webapp
3683b93a662aa25f324bac87daf388f1d08e879a070ef9ad7b31b4ca0ce6c0d7
jbl@pegasusio:~/docker-demo-webapp$ docker logs -f webapp

• So you see, the web app is running, now I id that To chekc if it was my Virtual BOx VM as a virtual device, that was rotten, but no :( :

Tomorrow I will quickly build a repo with a traefik 1.7 daemonset, which worked for me, my only problem with that deployement is that I could not configure it for https, but that 's another question.

I will also try your entire recipe by means of total despair, a least to get a working traefik.toml file that I tested.

I also eventually want to ask : is it me, or everybody is having a damn hard time trying to use Traefik ? (What was your experience, I mean, did you succeed on first try ? )

So thank you so much again for your and tty tomorrow

[Thoorium]

Kubernetes networking by itself can be quite challenging but thankfully, there is a lot of documentation and how-tos available to help with the challenges. Traefik adds another layer of complexity above this and requires knowledge on both levels to get working properly. When I first setup my cluster with Traefik, I was using Traefik v1 and the documentation for Kubernetes was...minimal. Thankfully a few brave souls (cited in my sources) figured some of the issues and I was able to piece everything together. At some point I moved to Traefik v2 and while the documentation was still rough, I was able to get it to work pretty fast. Now the Traefik Kubernetes documentation is a lot better.

Anyway, if I have the time tomorrow, I'll try to get Traefik to work without a LoadBalancer using the ClusterIP mode.

[Jean-Baptiste-Lasselle]

update :

• I appears that If I have a pending External IP on my LoadBalancer Service (the one you try with ClusterIP), it is because I use minkube, and minkube does not support LoadBalancer type, apparently.
• I tried using same LoadBalancer on AWS, with a pulti node cluster, and i got an external IP, and could access to the traefik dashboard, through the traefik ingress. And lso could deploy the apps

So I'll handle drifting to a more prod like K8s cluster for my home tests, but I am dying to ask you jst one question if that is okay :

What configuration do I have to apply to your recipe, so that i can deploy HTTPS apps ?

(thank you so much for making me able so far, to do what I already did, using Traefik v1.7)

For example, in my tests, I used a configmap to pass on a custom traefik config file, and I heard about the dynamic config new concept, I have a very hard time with it, but could not ever get it to let me serve https apps ...

[Thoorium]

Kubernetes without a LoadBalancer to provide IPs is not pretty. This is where metallb comes into play. Have you tried to apply my metallb configuration to your setup?

For HTTPS, I didn't try it but this should expose it via Traefik.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: traefik
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik-ingress-controller
  namespace: traefik
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:v2.0
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: admin
          containerPort: 8080
        args:
        - --api.insecure
        - --accesslog
        - --providers.kubernetescrd
        - --entrypoints.web.Address=:80
        - --entrypoints.websecure.Address=:443
---
kind: Service
apiVersion: v1
metadata:
  name: traefik
  namespace: traefik
  annotations: {}
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - protocol: TCP
    port: 80
    name: http
  - protocol: TCP
    port: 443
    name: https
  type: LoadBalancer
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - port: 8080
    name: dashboard

[Jean-Baptiste-Lasselle]

@Thoorium thank you so much !

• I will try your setup, with networking, one of my main gaols is to paly with entworking, in kubernetes (about containers, that's the last part where I want to get stronger)
• I will try the setup with https, and will feedback to you
• just to say, what I wanna try is vxlan,, later

[Jean-Baptiste-Lasselle]

couldnot resist before sleep : https://github.com/pokusio/the-traefik-path/releases/tag/0.0.1

• you will read there how I get the IP Address assigned by AWS to access my traefik in my cluster. I was looking for the Ingress everywhere, but realized that for every service, of type LoadBalancer, Kubernetes assigns an Ingress to it, automatically, without mentioning explicitly that Ingress among the Kubernetes Ingresses...
• That's weird, do you have any idea why they have two different ways of seeing what an Ingress is...? Or is it that a service of type LoadBalancer, actually becomes an Ingress, under the hood ...? Just really late thoughts
• I will definitely as soon as possible try your setup, I want to play more with k8s, especially network

[Thoorium]

While Kubernetes can run everywhere, it was built for cloud services primarily. As such, IP provisioning is done via LoadBalancer services which are external to Kubernetes and managed by the cloud providers. You can read a bit more here https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/. Since you are using a bare metal solution here, you need something to provide IPs to your services since you don't have an external load balancer to do the task. This is where MetalLB https://metallb.universe.tf/ comes to help. It is a LoadBalancer implementation for bare metal Kubernetes installations.

Using my setup will probably solve all your issues at this point.

[Jean-Baptiste-Lasselle]

what I am number interestde in :

• I want the same load balancer, but nw I have 3 master nodes , and 3 worker node, 6 virtual machines
• and I want to do it with the kind of networking you use : because no way will it just be master nodes IP load balancing,
• probably flannel assigns the External IP for a ClusterIP type service, because it is internal networking, not external (so pure flannel)
• So I am asking : did you manage to switch your flannel network , to an underlay network, using something like openvswitch ?

[Thoorium]

For the first 3 points, my setup will be able to do that. MetalLB handles the IP stuff between the router and the nodes. So if a node goes down, everything should get reassigned automatically and available again via the same IP.

For the last point, I've tested my setup with Kube-router instead of Flannel but that's pretty much it. I haven't tested any other network setup.

[Jean-Baptiste-Lasselle]

hi @Thoorium ....(but you speak French :) ? Let's keep writing in English so non-French can get into the discussion ?)

• Definitely will try you setup to experiment first 3 points at home on a "real" cluster
• Very interesting : I read yesterday, about Metallb , and it specifically mentions a difficulty with kube-router
• I am thrilled by the conversation : we have here a grasping point from where we can hard test on network, and the question is :
  • How do people set up a proper Kubernetes cluster, and get an External IP that actually routes into the kubernetes cluster, to a LoadBalancer Service Type ?
  • What I understand from your setup, is that you actually have all this "as long as you are in the flannel network".
  • So we will get our answer, if we can plug the flannel network, to any arbitrary network we choose, won't we ?
  • Meaning we need a Layer 2 connection, not just the router (kube-router ?) , what do you think ?

The work plan I have in mind :

• Do that connection of the flannel network, to any network, using a maniual test setup, anything dirty that will just confirm the mechanism
• Find how they do the exact same with OpenStack on premises, and Kubernetes provisioned inside OpenStack on prem
• Can't wait to get there, cause I finally want to try SDN n OpenDaylight there
• Kind of feel like proposing we collaborate to do that neatly, and share on github ?

[Jean-Baptiste-Lasselle]

hi @Thoorium :

• I will work on our subject this week end
• ok so now I understood :
  • When you create a kind: Service of type LoadBalancer, in Kubernetes, well you need an actual Load Balancer added to Kubernetes : and that's the Load Balancer who "gives" an External IP
  • the Kubernetes External DNS, will only help configuring with domain names resolution in a cluster context, noting like attribution of External IPs, though related, we understand why.
  • So your Metallb will perfectly do, and If I am right here in what I understand of Kubernetes, ten I'll say that in your case, you do, get a value under External IP, if you create a kind: Service of type LoadBalancer, am I right ?
  • And the External IP you get, lives in the flannel network
  • So all we need to know now, is how to connect Flannel to the outsite wrold, through a router.

[Jean-Baptiste-Lasselle]

hi again @Thoorium :

• withdraw Flannel from youyr recipe, and guess what you are going to get ? Well you will get Externel IP from the local network you VMs are using. SO It's a bridged networking to underlay network, isn'it?

• So now the most interesting setup is (reproducing AWS / Cloud Providers setup^inside datacenters) :

  • We use a VXLAN networking solution like Flannel, or mininet , or whatever we will find in the future, for VX LAN netgworking in Kubernetes
  • We set up routers, that connect the VXLAN, to the underlying machine of our choice, in our infrastructure.

• This way, we have network isolation, that acts as multiple successive airlocks

• Any machine we choose, in our infrastructure :

• Here above :

  • The two clouds, are two separate OpenStack relying on their own underlay infra (hardware)
  • The green network, is the VXLAN network I am talking about in previous bullet points
  • The glowing pink network, is the underlay network
  • The two glowing pink ovals are kubernetes cluster, both using Metallb :
  • the Zero Chamber uses Metallb with default netwoking : it is going to give IP addresses to LoadBalancer type Services' External IP, from the underlay network, connected to the outside world
  • and it is going to route using the setup

• where the two networks are connected, is where we have the VMs with an OpenVSwitch switch running inside, with both green and glowing pink network interfaces (connected to the OpenVSwitch switch). :

  • on those machines we have war zone security, like everything that we can think of IDS, firewall, (the whole f army)**
  • those machine are running OpenVSwitch switches, rochestrated by OpenDAylight, or any other SDN controller, and they together, constitute the DATA PLANE, of the SDN architecture. Those machiens are runnign a typical stateless service (dumb switch), wihch can be scaled in a kuberenetes cluster, with metallb
  • The CONTROL PLANE, can consist of another fleet of machines, running the control palne, that is, the SDN Controller : A Scallable StatefulSet in Kubernetes, also with metalLB
  • those data and contraol plane, as saclaable units, will need a Kubernetes NodeAutoScaler wich will create / delete VMs with either switches, or sdn controller, to sacle up or down, the Kuerbentes CLuster capacity, beyond pod density limit.

• We don't care on what machines switch and routers run (containers, VMs, or real hardware) : what matters is to what network interfaces they are connected

• The setup in my schematics brings in a little bit of network redundancy, in case of a nuke.

  • What is very important here, is that I want that stup, to teswt out BGP, with MetallB, and between the two OpenStack installations (the two big clouds)

• What is funny though here, is that I am problably provision the OpenStack ... inside containers, using RDO

[Jean-Baptiste-Lasselle]

So I propose

If you are interested, I would like to test every single assertion in https://www.youtube.com/watch?v=Ytc24Y0YrXE

• example :

  • Let myClusterA be a Kubernetes cluster, installed using [TODO: give exact reference to provisioning recipe] , having 6 nodes 3 masters, 3 workers, using k3s/k3d
  • Say :
  • in myCLusterA, I deployed metallB as LoadBalancer, directly bridged to my underlay network
  • we deploy Traefik v1.7 as IngressController in myClusterA,
  • we deploy two apps myFirstApp, and myOtherApp, in myCLusterA, each endowed with an Ingress for traefik, like https://github.com/containous/traefik/blob/v1.7/examples/k8s/cheese-ingress.yaml

• I do all this in a network that is behind a router, behind my home internet access router : that, so I can entirely manage my network IP address space, without having to reconfigure my home's ISP router

• behing that router, my local network IP Address space will be 192.168.3.0/24, and when I will have two openstacks, I will use two networks with 192.167.0.0/16 and 192.168.0.0/16 IP address spaces for each openstack installation. I will connect those two networks to one root network, from which I'll operate the two others, a smaller network, where me, the super admin devops, works safely protected from the internet, behind firewalls, 192.169.0.0/24.

• Then answer those questions :

  • when my apps deployed, to which cluster node VM, does the External IP granted to the Ingress, belongs ? (A master only ? Who creates that IP address, and gives it to that machine? How can those IP adresses be any other IP address, than the IP of one of hte Cluster nodes ? see flannel)
  • where do those elastic ips come from, in aws eks ? IS an AWS ELB, actually an EC2 machine group, of 2 machines, with 2 pulblic IPs, and the L4 load balancer running inside of it (and then those IPs are routed back to at least one of the machines that are Kubernetes cluster nodes ? )

• Then test (automated tests) if :

  • the Ingress for myFirstApp is listening on all cluster nodes (or only masters ?) (the app has an Ingress, but it kind: Service is of type ClusterIP, not LoadBalancer.)
  • the Ingress for myOtherApp is listening on all cluster nodes (or only masters ?)

• Do same test, but this time, it is about Traefik v2, and IngressRoutes, instead of

[Thoorium]

I appreciate the interest but this is getting out of scope from the initial question. I would suggest that you create a new repository and document your findings/advancement there instead ;)

You can copy/paste this comment and I'll answer the questions there at the best of my abilities.

[Jean-Baptiste-Lasselle]

I agree :) : I'll be glad to collaborate with you here : https://github.com/pokusio/k3s-topgun