Closed TrentonAdams closed 5 years ago
I apologize, this is not true, it does not work in kubeadm-dind either. I don't know how I had it working.
FWIW ingress-nginx actually uses kind for CI, I'll dig into this a bit before reaching out to them if need be ...
I apologize, this is not true, it does not work in kubeadm-dind either. I don't know how I had it working.
Ah, ok. Good luck with that!
I decided to install and use helm to get it running. It then gave me instructions, and the proper ingress yaml to use.
For my application, this now means a proper load balancing of back-ends...
$ for i in {1..10}; do curl http://api.example.com:30076/health/node; echo; done
inotes-f759d4b66-2rvhx
inotes-f759d4b66-fmzdb
inotes-f759d4b66-qbqng
inotes-f759d4b66-qbqng
inotes-f759d4b66-2rvhx
inotes-f759d4b66-q5c8h
inotes-f759d4b66-q5c8h
inotes-f759d4b66-qbqng
inotes-f759d4b66-wvwfc
inotes-f759d4b66-fmzdb
So I got the ingress going on kind, but doing it via helm doesn't work at all. it claims about not having the permissions.
Is this a misunderstanding on my part for not properly configuring kind, or should I file a bug regarding this?
helm install stable/nginx-ingress --name my-ingress --namespace ingress
Error: release my-ingress failed: namespaces "ingress" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "ingress"
Maybe you need read https://kubernetes.io/docs/reference/access-authn-authz/authentication/ and https://helm.sh/docs/using_helm/#role-based-access-control
Your helm
looks incorrectly configured.
Oh, I didn't realize kind had authorization stuff enabled. I've been using kubeadm-dind, so I never had an issue with it before. I suppose I need to investigate that a little. Thanks!
Ah yeah that'll do it. FWIW we're using kubeadm defaults for auth. I would expect recent clusters to have RBAC by default, kubeadm or otherwise, but this didn't even exist in earlier versions of Kubernetes 😅
Odd, I never had a problem using helm with kubeadm-dind, perhaps they didn't include it by default.
You can skip it by using Helm v3, or using Helm v2 with tiller run local (one hack way).
I'm using helm3 but I think it's not possible to use stable/nginx-ingress
chart because it uses LoadBalancer which is not supported by the kind
:
PS C:\> helm install my-ingress stable/nginx-ingress
...
PS C:\> kubectl get all
NAME READY STATUS RESTARTS AGE
pod/my-ingress-nginx-ingress-controller-58bf5b95-8s8js 0/1 Running 0 51s
pod/my-ingress-nginx-ingress-default-backend-7f649fbcb6-xkgc4 1/1 Running 0 51s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 112s
service/my-ingress-nginx-ingress-controller LoadBalancer 10.97.191.140 <pending> 80:30298/TCP,443:31142/TCP 51s
service/my-ingress-nginx-ingress-default-backend ClusterIP 10.96.128.194 <none> 80/TCP 51s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/my-ingress-nginx-ingress-controller 0/1 1 0 51s
deployment.apps/my-ingress-nginx-ingress-default-backend 1/1 1 1 51s
NAME DESIRED CURRENT READY AGE
replicaset.apps/my-ingress-nginx-ingress-controller-58bf5b95 1 1 0 51s
replicaset.apps/my-ingress-nginx-ingress-default-backend-7f649fbcb6 1 1 1 51s
Notice that LoadBalancer is just pending, because it's not supported by kind
.
Did someone get this working? I'm trying to make a proof of concept script running nginx ingress on kind.
@Ciantic That's normal for LoadBalancer to not be supported, it's a cloud only feature. Ingress controllers can do load balancing too, but LoadBalancer is reserved for cloud use.
https://kubernetes.io/docs/concepts/services-networking/
LoadBalancer: Exposes the Service externally using a cloud provider’s load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.
@TrentonAdams Okay I know that. But how are we supposed to use
helm install my-ingress stable/nginx-ingress
If it creates a LoadBalancer?
I tried to do switcheroo:
kubectl patch svc my-ingress-nginx-ingress-controller --type=json -p '[{"op": "replace", "path": "/spec/type","value":"NodePort"}, {"op": "replace", "path": "/spec/ports/0/nodePort", "value": 30080}, {"op": "replace", "path": "/spec/ports/1/nodePort", "value": 30443}]'
But it gives error:
The Service "my-ingress-nginx-ingress-controller" is invalid:
* spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
* spec.ports[1].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
I can however patch it to ClusterIP if I remove the ports, but isn't the only option to convert that LoadBalancer to a NodePort in order to use nginx ingress with kind
?
@Ciantic using --set controller.service.type=NodePort
(default is LoadBalancer)
@aledbf thanks, that works.
For posterity, I'll post the whole working example (in Powershell):
# Setup
kind create cluster --config kube-config.yaml
$env:KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
# https://github.com/helm/charts/tree/master/stable/nginx-ingress
helm install my-ingress stable/nginx-ingress --set controller.service.type=NodePort --set controller.service.nodePorts.http=30080 --set controller.service.nodePorts.https=30443
kubectl create -f service.yaml
# Wait for nginx to come online (120 seconds might not be enough)
timeout.exe 120
curl.exe http://localhost:30080/test
Pause
# Teardown
$env:KUBECONFIG=""
kind delete cluster --name kind
kube-config.yaml
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30080
hostPort: 30080
service.yaml
---
apiVersion: v1
kind: Service
metadata:
name: srvc-nginx
spec:
ports:
- port: 80
selector:
srvc: nginx-selector
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: deployment-nginx
spec:
template:
metadata:
labels:
srvc: nginx-selector
spec:
containers:
- name: working-nginx
image: nginxdemos/hello # https://github.com/nginxinc/NGINX-Demos/tree/master/nginx-hello
ports:
- containerPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: example
spec:
rules:
- host: localhost
http:
paths:
- backend:
serviceName: srvc-nginx
servicePort: 80
path: /test
# This section is only required if TLS is to be enabled for the Ingress
# tls:
# - hosts:
# - www.example.com
# secretName: example-tls
# If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
# ---
# apiVersion: v1
# kind: Secret
# metadata:
# name: example-tls
# namespace: foo
# data:
# tls.crt: <base64 encoded cert>
# tls.key: <base64 encoded key>
# type: kubernetes.io/tls
@Ciantic Using a single node as per you example it works just fine.
I added additional worker nodes and it works only if the pod has been deployed onto the node where I configured the extraPortMappings.
I tried basic examples (pod+service) using nodePort: they work regardless the node where the pod has been deployed (as expected according to the nodePort concept).
What am I missing?
I'm using kind on MacOs
kind not properly supporting ingress-nginx
Keep in mind this worked in kubeadm-dind
What happened: I installed the ingress-nginx as indicated at https://kubernetes.github.io/ingress-nginx/deploy/ use the mandatory file and the "baremetal" file, but any requests to my app return it's home page, rather than proper sub-paths
What you expected to happen: I expected to have sub-paths passed to the back-end app
How to reproduce it (as minimally and precisely as possible):
I then ran a sample app...
I then applied this ingress rule...
Subpath returns the following when ran in just docker (not kind)
Anything else we need to know?: Nope, It should just work.
Environment:
Docker version: Docker version 18.09.6, build 481bc77
OS (e.g. from
/etc/os-release
):