poseidon / typhoon

Minimal and free Kubernetes distribution with Terraform
https://typhoon.psdn.io/
MIT License
1.93k stars 320 forks source link

Cannot run AWS nginx ingress controler addon #1508

Closed bruvio closed 1 week ago

bruvio commented 2 weeks ago

Description

after running a vanilla cluster, installing the nginx-ingress

with kubectl apply -f .terraform/modules/bruvio/addons/nginx-ingress/aws and rbac kubectl apply -f .terraform/modules/bruvio/addons/nginx-ingress/aws/rbac

i run a simple ingress nginx

and geting a 404

I see no errors in the pod

kubectl logs nginx-deployment-7cdbd54bcf-sjg4q                                                                                                                                                                   [31/08/24|7:01:07|]
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/08/31 04:44:39 [notice] 1#1: using the "epoll" event method
2024/08/31 04:44:39 [notice] 1#1: nginx/1.27.1
2024/08/31 04:44:39 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2024/08/31 04:44:39 [notice] 1#1: OS: Linux 6.6.43-flatcar
2024/08/31 04:44:39 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/08/31 04:44:39 [notice] 1#1: start worker processes
2024/08/31 04:44:39 [notice] 1#1: start worker process 21
2024/08/31 04:44:39 [notice] 1#1: start worker process 22

and no erros for the controllers

kubectl logs nginx-ingress-controller-7d445ffd6c-xfxvn -n ingress                                                                                                                                                [31/08/24|7:00:53|]
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.5.1
  Build:         d003aae913cc25f375deb74f898c7f3c65c06f05
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W0831 05:00:43.815783       7 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0831 05:00:43.816076       7 main.go:209] "Creating API client" host="https://10.3.0.1:443"
I0831 05:00:43.822894       7 main.go:253] "Running in Kubernetes cluster" major="1" minor="31" git="v1.31.0" state="clean" commit="9edcffcde5595e8a5b1a35f88c421764e575afce" platform="linux/amd64"
I0831 05:00:43.908647       7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
W0831 05:00:43.929032       7 store.go:808] Unexpected error reading configuration configmap: resource name may not be empty
I0831 05:00:43.942474       7 nginx.go:260] "Starting NGINX Ingress controller"
I0831 05:00:43.947192       7 store.go:521] "ignoring ingressclass as the spec.controller is not the same of this ingress" ingressclass="nginx"
I0831 05:00:45.045660       7 store.go:426] "Ignoring ingress because of error while validating ingress class" ingress="default/example-ingress" error="no object matching key \"nginx\" in local store"
I0831 05:00:45.045698       7 store.go:426] "Ignoring ingress because of error while validating ingress class" ingress="default/nginx-ingress" error="ingress does not contain a valid IngressClass"
I0831 05:00:45.045713       7 store.go:426] "Ignoring ingress because of error while validating ingress class" ingress="default/simple-nginx-ingress" error="ingress does not contain a valid IngressClass"
I0831 05:00:45.142933       7 nginx.go:303] "Starting NGINX process"
I0831 05:00:45.143049       7 leaderelection.go:248] attempting to acquire leader lease ingress/ingress-controller-leader...
I0831 05:00:45.143947       7 controller.go:168] "Configuration changes detected, backend reload required"
I0831 05:00:45.146103       7 status.go:84] "New leader elected" identity="nginx-ingress-controller-78595d44df-vz479"
I0831 05:00:45.214678       7 controller.go:185] "Backend successfully reloaded"
I0831 05:00:45.214910       7 controller.go:196] "Initial sync, sleeping for 1 second"
I0831 05:00:45.214981       7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress", Name:"nginx-ingress-controller-7d445ffd6c-xfxvn", UID:"7e19e558-9474-46e5-bc22-8b449a3f4424", APIVersion:"v1", ResourceVersion:"50224", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

Steps to Reproduce

terraform cluster install nginx-controller from addon folder run nginx ingress deployment using

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: default
data:
  default.conf: |
    server {
        listen 80;
        server_name localhost;

        location /nginx {
            alias /usr/share/nginx/html/;
            try_files $uri $uri/ =404;
        }

        location / {
            root /usr/share/nginx/html;
            index index.html;
        }
    }

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        volumeMounts:
        - name: nginx-config-volume
          mountPath: /etc/nginx/conf.d/default.conf
          subPath: default.conf
      volumes:
      - name: nginx-config-volume
        configMap:
          name: nginx-config

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
  - port: 80
    targetPort: 80

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: mydns.net
    http:
      paths:
      - path: /nginx
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80

Expected behavior

curl http/mydns.net/nginx should return 200

Environment

Possible Solution

Link to a PR or description.

bruvio commented 2 weeks ago

I also tried using the guidance from AWS installing the controller using

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.34.0/deploy/static/provider/aws/deploy.yaml

and still getting same. can it be a infra bug?

bruvio commented 2 weeks ago

pretty sure the ingress is working because if i deploy a serive as nodeport with a swcurity group that open traffic on that port i can connect at mydns.net:PORT

bruvio commented 1 week ago

I installed the controller following

CHART_VERSION="4.4.0"
APP_VERSION="1.11.2"

mkdir ./manifests/

helm template ingress-nginx ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--version ${CHART_VERSION} \
--namespace ingress-nginx \
> ./manifests/nginx-ingress.${APP_VERSION}.yaml
kubectl create namespace ingress-nginx
kubectl apply -f ./manifests/nginx-ingress.${APP_VERSION}.yaml

then deployed

apiVersion: v1
kind: ConfigMap
metadata:
  name: service-a
data:
  path-a.html: |
    "/path-a.html" on service-a
  path-b.html: |
    "/path-b.html" on service-a
  index.html: |
    "/" on service-a  
  404.html: |
    service-a 404 page
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: service-a-nginx.conf
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;
    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;
    events {
        worker_connections  1024;
    }

    http {
        sendfile        on;
        server {
          listen       80;
          server_name  localhost;

          location / {
              root   /usr/share/nginx/html;
              index  index.html index.htm;
          }

          error_page 404 /404.html;
          error_page   500 502 503 504  /50x.html;
          location = /50x.html {
              root   /usr/share/nginx/html;
          }
        }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-a
  labels:
    app: service-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-a
  template:
    metadata:
      labels:
        app: service-a
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - name: html
          mountPath: "/usr/share/nginx/html/"
        - name: config
          mountPath: "/etc/nginx/"
      volumes:
      - name: html
        configMap:
          name: service-a
      - name: config
        configMap:
          name: service-a-nginx.conf
---
apiVersion: v1
kind: Service
metadata:
  name: service-a
spec:
  selector:
    app: service-a
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

--- 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: service-a
spec:
  ingressClassName: nginx
  rules:
  - host: app.mydns.net
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: service-a
            port:
              number: 80
---

and the service is not reachable.

I tested this with kind and using port-forward and a host file pointing at localhost works. So either I am doing something the infra does not support or there is something blocking traffic.

dghubble commented 1 week ago

This isn't related to Typhoon, which provides a working Kubernetes cluster and even an example for running an Ingress Controller.

As you say, the controller is running fine and if you're seeing a 404, that means the infra pieces are working.

A warning in the logs you've shown point to the likely issue in your example ingress object.

Ignoring ingress because of error while validating ingress class" ingress="default/nginx-ingress" error="ingress does not contain a valid IngressClass