search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.36k stars 4.88k forks source link

add integration test for ingress addon #6922

Open medyagh opened 4 years ago

alanbrent commented 4 years ago

(Continuing from a #minikube Slack convo)

At a high level, we do the following at our org:

  1. Install an Ingress Controller (ours is haproxy for no good reason, any Ingress Controller will do including I assume a minikube ingress addon)
    • we do this as part of a wrapper script that prepares the local environment, starts minikube, and installs core components (e.g. Ingress Controllers)
  2. Add an Ingress object to the manifest(s) for a given application that we want to be externally available
    • the Ingress object would be configured to route traffic to a Service object that fronts the Pod(s) running the application

So again at a high level, one could test this by standing those things up and then issuing a curl -H 'Host: $APP_SVC_NAME.$K8S_NAMESPACE.svc.cluster.local' $MINIKUBE_IP/test_path_eg_healthz.

It's not a script, but hopefully the high level description is useful.

medyagh commented 4 years ago

@alanbrent could you share the yaml files and all the configurations as you do it manually to setup ha proxy ?

alanbrent commented 4 years ago

(Updated w/ Namespace and default backend bits)

Here you go. Note that the TLS bits aren't actually sensitive, it's just a self-signed cert I generated to satisfy whatever cargo culting I was doing :)

Very simple (but now quite dated) `haproxy-ingress` stack ```yaml --- ######################################################################################################################## ## NETWORK-RELATED MANIFESTS ######################################################################################################################## apiVersion: v1 kind: Namespace metadata: name: ingress-controller --- apiVersion: v1 kind: Service metadata: name: "default-http-backend" labels: k8s-app: "default-http-backend" environment: "development" namespace: ingress-controller spec: ports: - port: 80 targetPort: 8080 selector: k8s-app: "default-http-backend" environment: "development" --- ######################################################################################################################## ## RBAC-RELATED MANIFESTS ######################################################################################################################## apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: "haproxy-ingress" rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "extensions" resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: namespace: "ingress-controller" name: "haproxy-ingress" rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get - create - update --- apiVersion: v1 kind: ServiceAccount metadata: namespace: "ingress-controller" name: "haproxy-ingress" labels: app: "haproxy-ingress" environment: "development" --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: "haproxy-ingress" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: "haproxy-ingress" subjects: - kind: ServiceAccount name: "haproxy-ingress" namespace: "ingress-controller" --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: "haproxy-ingress" namespace: "ingress-controller" roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: "haproxy-ingress" subjects: - kind: ServiceAccount name: "haproxy-ingress" namespace: "ingress-controller" --- ######################################################################################################################## ## CONFIGURATION-RELATED MANIFESTS ######################################################################################################################## apiVersion: v1 kind: ConfigMap metadata: name: "default-http-backend" labels: k8s-app: "default-http-backend" environment: "development" namespace: ingress-controller data: # This configMap is necessary because the lower.case names fail the test regex for container environment variables in the Kubernetes code cluster.name: "development" bootstrap.memory_lock: "true" ES_JAVA_OPTS: "-Xms512m -Xmx512m" --- apiVersion: v1 kind: ConfigMap metadata: namespace: "ingress-controller" # Namespace is created in default-http-backend manifest (different folder) name: "haproxy-ingress" labels: app: "haproxy-ingress" environment: "development" data: ssl-redirect: "false" # No TLS in local dev stats-port: "19080" # Matches what we use in production, though without the /hap_stats uri --- # This is just a self-generated cert/key pair. TODO: do this programatically or probably just disable it apiVersion: v1 kind: Secret metadata: namespace: "ingress-controller" name: tls-secret labels: app: "haproxy-ingress" environment: "development" type: kubernetes.io/tls data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJRENDQWdpZ0F3SUJBZ0lKQUlScUVDM1NRdW9GTUEwR0NTcUdTSWIzRFFFQkJRVUFNQlF4RWpBUUJnTlYKQkFNVENXeHZZMkZzYUc5emREQWVGdzB4TnpBNU1qSXlNak13TkRKYUZ3MHhPREE1TWpJeU1qTXdOREphTUJReApFakFRQmdOVkJBTVRDV3h2WTJGc2FHOXpkRENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DCmdnRUJBTWxPWXdoYUpIczk4QUl1ekdmdlVhMGJlb1hNYmtVSXozcWFNbHlCZGJzbUNOejB6VXdWdVhQUDV0ZHUKNm56dHNMVWNnVWtUYTNyMlQ1ei9lSnhYQmFyVmUvZStZdlVUZ01zdHlxVU5ZTlNpUDFMOTNkVXYxckw5c2tSKwp0ZHlRNnRqSWFXOXZpZTZVWEd5R1JSMXZNVXVmc1poTUtDTEcrRy9SN21vR29TZlI5Lzdzc2I3SVk1T2VNb3MwCjNPZkhvNkZWRk5wRSs3MUlJaVZHZUM3d0VtSnZYWmlLRXI0dVlBVjZqdUxYcTVvUklkZTVnZDRLZXhvbE03TVcKMERxY1JRQU9ZVHRsOSs5UjFCN2tJNmZTWUl6eU9FWitqL3VjQ29JRkJYR0MwZFdhQUhOUW1JaEJ5aHg5YmJEZgpDMXlJR0grUTl3RmlzSnJzaFZsdEE4QWF3WkVDQXdFQUFhTjFNSE13SFFZRFZSME9CQllFRkI4OVA4VGtmTUNYCjZVdWxXSUI1bEpMek50YVdNRVFHQTFVZEl3UTlNRHVBRkI4OVA4VGtmTUNYNlV1bFdJQjVsSkx6TnRhV29SaWsKRmpBVU1SSXdFQVlEVlFRREV3bHNiMk5oYkdodmMzU0NDUUNFYWhBdDBrTHFCVEFNQmdOVkhSTUVCVEFEQVFILwpNQTBHQ1NxR1NJYjNEUUVCQlFVQUE0SUJBUUJ0MTBhYUExMWxDMDVZNVZUaFJTRGFsVlpycmNMMjlPbkQyYVgwClRITW0yT0x6bHROVGhvaU9ueGZhRE9WK24ydWRGMzFtT1VXZnV5aHJXTzhYTFllcmVqUXF6VGFFOXhac0k1VW8KVnA5bktVTFNZd0M0Z2IyTXdoZmtRNVl6YXZNZFAra1I2cU8xWUpWMmRWd1RhMTR2RGxaYytvaWJ6UStiTmtWagpzT2NxWjdaTU5tNEpCYWxUOUlHdUdKa0lFUjhLZ1VKMUFUcVhTM1NWUm9Ba0ZieXBaMkZNd2dhTFNpMHNWMFdKCkl1b1FjZGlTRWY5bGdTeU90OTZNRUJWT1ZTQlYrZmZOTENWNGt5ZmNFemdoWEpna3pTcUZRQ0ZRcUp4WGVsangKNTAzdG9ramNRbGcyWC8yQ1VTZmVQYndleDFMTmtXSlhUSmxMYWlVdFczOHdHUzA2Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBeVU1akNGb2tlejN3QWk3TVorOVJyUnQ2aGN4dVJRalBlcG95WElGMXV5WUkzUFROClRCVzVjOC9tMTI3cWZPMnd0UnlCU1JOcmV2WlBuUDk0bkZjRnF0Vjc5NzVpOVJPQXl5M0twUTFnMUtJL1V2M2QKMVMvV3N2MnlSSDYxM0pEcTJNaHBiMitKN3BSY2JJWkZIVzh4UzUreG1Fd29Jc2I0YjlIdWFnYWhKOUgzL3V5eAp2c2hqazU0eWl6VGM1OGVqb1ZVVTJrVDd2VWdpSlVaNEx2QVNZbTlkbUlvU3ZpNWdCWHFPNHRlcm1oRWgxN21CCjNncDdHaVV6c3hiUU9weEZBQTVoTzJYMzcxSFVIdVFqcDlKZ2pQSTRSbjZQKzV3S2dnVUZjWUxSMVpvQWMxQ1kKaUVIS0hIMXRzTjhMWElnWWY1RDNBV0t3bXV5RldXMER3QnJCa1FJREFRQUJBb0lCQVFDSnF2elpFbVNPais0YQpZQXpVSzN0czZqVGhXY3d6ay9Ib1E1bWJYOEcyNml6M2hzYjdlSVBxYXN6UW1mN3N6dFNoRjB4blFFdXB2TmIzClpGTUVnQkxWSU1oMXhoTUF6WVN2N3JIK0xHY0RJNjR3d0lLN3I4cWhnQlF3K1lLL3c2K0g3NDhEY1lFZjB3QmUKQzNrcXh3Y3NVNXJ0cjI5blVURkhyNGJmcFljQ3BrUHl3cHlocWQwQU1GVUQ2bWhYbDhyaSs0NHRCVVo0NWxSdgpla2RySGNzMCsweGdXa3dOejJUeDRYN1hIcWVGSVI0RnpYdTBVNTNZL3l1TGZlaU1GRWF5Q2NBYnZVd1B0NVVXCm9IbFdmUHlCR2swOElwU2NFNThYV05XeUFkRVFhRVliK3NEMXhtTGFLdlVqc2kzLzBUVTNTOTlveUFVcHJjWVQKUUNGNU5veFJBb0dCQU9VOXVtNnNsWDdTakRGVkxiY2VBcVlmQ0hzL1RHTWp1SWdnd3MyV1JoSTlMN0oyUzd1QQoxeW1lZnJkNFJ5NHVuQ3QvT2hISllVcFpvaTZmVUdqNmR3UEpEa1ZDQVZNNHRyZkN2YUpmRmpSYlc1c2sxeU53CktnMzZ5RHVEUUVpVkM1bEFDR0VIblhXT3VmWXdvWWNheVVhR3gvMGZEV28vRk82Mjdza2FodWNGQW9HQkFPRE4KNVhWQ2VQOUhuMjljNS9uNzM0bEpCNURHK1FLZldTTVpCWVBJTDRDSnZZY0FwSHBkOFFGOTRVNUZXc0U0aHJtdQpNaG9jejFWUTJ5OWdJbU1PNGJnTjJBb3E0bFdIbC9WOVVscnQzbTZ6dnVSVlF3RlVTOEFSZ1VTZ3pkcVNHTTNvCjIzZ1BpSnRtaXBBL0tGSlhRc09aRHpkeFhmeGFFOW1ZUXcrOHdCNGRBb0dCQU5jYzk2Y0hPRG1MNVpZZGNncVYKbExMQ0NEbHZDb1UxYmVIL3JBMHphSW1ROVFNb1VxU2VvRWtqdTVENnd3Q3BLRThBMGtNQ2o5TXZEU3Rtd0JQaQpoRy8rNk85QW92R3ZGSGFYdlU5eW51bCttUldhbW05Y3Yzbml6S1piNGFTNmgrbkUxcGdHaDFGYkgxSFFES00xCkZZNml4MlBWRWNlVGJsY2JQak5rT3hicEFvR0JBTXBwZVg2QVVMUUNTRk1jUXg4Nkk0YllROENlOTF6ZUg0TlQKcHp5R0JnZGxma3BOVU11QlJGeUVmRlplYWgzcUNKeXRkUjAvN1lzekYzZnJ4bXFNM3I0N2NOZU96MllWaGNTYgpoNGlwL3dxTU80Z2FJUS9VVjdSanpwSDllL3llYVVrVnJtN0hCY2h1cHJyRXMwaVltMXZFdUFmQ3dzNVNTT2MyCmlzTmp2MVFkQW9HQWVLOHlXUWJ6Qng5YTZBWTNDY3JyTHUvSTJtY2RrdUQ0YnFwOU00TlZSVWlBUFhqSjJ0eGkKOURNWkdQWGs1bFdOaTdRMVVSaVpGeDhPb3FYcmgrcUo1MDAyOFhWRHlsd2p2M2ZwWkJLYnluOHhoU0hJbysyUAp4UWorVDBLYkN4NXlyTXpMU2hhTklGMWxmZnJEY2FybHZYNHZ6ZW1UOUVDYW5QOTNvV3BSaE1VPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= --- ######################################################################################################################## ## WORKLOAD-RELATED MANIFESTS ######################################################################################################################## apiVersion: apps/v1 kind: Deployment metadata: name: "default-http-backend" labels: k8s-app: "default-http-backend" environment: "development" namespace: ingress-controller spec: selector: matchLabels: k8s-app: "default-http-backend" environment: "development" template: metadata: labels: k8s-app: "default-http-backend" environment: "development" spec: terminationGracePeriodSeconds: 0 containers: - name: "default-http-backend" # Any image is permissable as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: "gcr.io/google_containers/defaultbackend:1.0" livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi --- apiVersion: apps/v1 kind: DaemonSet metadata: name: "haproxy-ingress" labels: app: "haproxy-ingress" environment: "development" namespace: "ingress-controller" annotations: kubernetes.io/ingress-class: "haproxy" spec: selector: matchLabels: app: "haproxy-ingress" environment: "development" template: metadata: labels: app: "haproxy-ingress" environment: "development" spec: hostNetwork: true serviceAccountName: "haproxy-ingress" containers: - name: "haproxy-ingress" image: "quay.io/jcmoraisjr/haproxy-ingress:snapshot" imagePullPolicy: IfNotPresent args: - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --default-ssl-certificate=$(POD_NAMESPACE)/tls-secret - --configmap=$(POD_NAMESPACE)/haproxy-ingress - --reload-strategy=multibinder ports: - name: http containerPort: 80 - name: https containerPort: 443 - name: stat containerPort: 1936 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace --- ```

I would imagine that constructing your tests around the ingress-controller minikube addon would be a much easier path to achieve the desired testing, but I may not fully understand what you're looking to achieve here :)

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

sharifelgamal commented 4 years ago

/lifecycle frozen