Open Roxyrob opened 9 months ago
As written in the referenced haproxy ingress controller issue, Until this issue is identified and resolved I found a WORKAROUND that seems stable enough.
Following this link: Haproxy document SSL / TLS
Manually create a certificate
openssl req -x509 -newkey rsa:2048 -keyout example.key -out example.crt -days 365 -nodes -subj "/C=US/ST=Ohio/L=Columbus/O=MyCompany/CN=example.com"
Manually create a secret
kubectl create secret tls -n {namespace} haproxytempcert --cert="example.crt" --key="example.key"
Temporary Change haproxy ingress ConfigMap
kubectl edit configmap -n {namespace} {configmap_name} apiVersion: v1 kind: ConfigMap metadata: name: ... namespace: {namespace} data: ... ssl-certificate: "{namespace}/haproxytempcert" <=== (Add)
Doing this "sometimes" seems "to wakeup" ingress controller that create default cert using haproxytempcert. Default certificate now exists and contains my Self Signed certificate.
If ingress controller DOES NOT wake-up, leave the haproxytempcert secret and ConfigMap annotation (ssl-certificate) to be used by haproxy ingress controller.
If Yes now you can delete temporary secrets and ConfigMap annotation:
Delete "ssl-certificate" from ConfigMap
kubectl edit configmap -n {namespace} {configmap_name} apiVersion: v1 kind: ConfigMap metadata: name: ... namespace: {namespace} data: ...
kubectl delete secret -n {namespace} haproxytempcert
Default certificate survive and now contains default haproxy Self-Signed certificate. What one expect from start.
Better to know why and where this happen.
@Roxyrob Can you try with the following custom settings, where controller.defaultTLSSecret.secretNamespace
refers to your custom namespace?
controller:
defaultTLSSecret:
enabled: true
secretNamespace: namespace
@dkorunic, was the first try.
Indeed deployment manifest has "--default-ssl-certificate" correctly set but no default certificate was created anyway:
apiVersion: apps/v1 kind: Deployment metadata: annotations: ... labels: app.kubernetes.io/instance: ingress-haproxy-internal app.kubernetes.io/name: kubernetes-ingress app.kubernetes.io/version: 1.10.10 helm.sh/chart: kubernetes-ingress-1.35.5 name: ingress-haproxy-internal namespace: ingress-haproxy-internal spec: progressDeadlineSeconds: 600 replicas: 2 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/instance: ingress-haproxy-internal app.kubernetes.io/name: kubernetes-ingress ... template: metadata: creationTimestamp: null labels: app.kubernetes.io/instance: ingress-haproxy-internal app.kubernetes.io/name: kubernetes-ingress spec: containers: - args: - >- --default-ssl-certificate=ingress-haproxy-internal/ingress-haproxy-internal-default-cert - '--configmap=ingress-haproxy-internal/ingress-haproxy-internal' ...
When deployment do not create default certificate the only way is the WORKAROUND above.
I haven't been able to reproduce your issue, it works for me without any issues. Configuration override used:
controller:
defaultTLSSecret:
enabled: true
secretNamespace: testspace
Helm command to install:
$ helm install testissue216 --create-namespace --namespace testspace -f myvalues.yaml ./kubernetes-ingress
Verify secrets:
$ kubectl get secrets --namespace testspace NAME TYPE DATA AGE
sh.helm.release.v1.testissue216.v1 helm.sh/release.v1 1 30s
testissue216-kubernetes-ingress-default-cert kubernetes.io/tls 2 29s
$ kubectl describe secret testissue216-kubernetes-ingress-default-cert --namespace testspace
Name: testissue216-kubernetes-ingress-default-cert
Namespace: testspace
Labels: app.kubernetes.io/instance=testissue216
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kubernetes-ingress
app.kubernetes.io/version=1.10.10
helm.sh/chart=kubernetes-ingress-1.36.0
Annotations: helm.sh/hook: pre-install
helm.sh/hook-delete-policy: before-hook-creation
Type: kubernetes.io/tls
Data
====
tls.crt: 1159 bytes
tls.key: 1679 bytes
Check IC logs:
$ kubectl logs testissue216-kubernetes-ingress-d86cf687d-jkwjn --namespace testspace| grep certificate
2024/01/16 16:04:54 Default ssl certificate: testspace/testissue216-kubernetes-ingress-default-cert
Everything looks correct. To me it looks like you haven't matched secretNamespace
and namespace
while installing your Ingress Controller instance. Sadly due to how Helm renders values from values.yaml we cannot match that automatically, it has to be specified by hand.
In fact, I was wrong in thinking that we need to match controller.defaultTLSSecret.secretNamespace
, it contains '{{ include "kubernetes-ingress.namespace" . }}'
which will correctly render into string with {{ tpl .Values.controller.defaultTLSSecret.secretNamespace . }}
.
As shown below it works as expected, no visible issues whatsoever. Required ingress controller service arguments are enabled, default SSL certificate is being created and it gets exported to the filesystem as needed:
$ helm install testissue216 --create-namespace --namespace testspace2 -f myvalues.yaml ./kubernetes-ingress
$ kubectl get secrets --namespace testspace2
NAME TYPE DATA AGE
sh.helm.release.v1.testissue216.v1 helm.sh/release.v1 1 2m54s
testissue216-kubernetes-ingress-default-cert kubernetes.io/tls 2 2m54s
$ kubectl describe secret testissue216-kubernetes-ingress-default-cert --namespace testspace2
Name: testissue216-kubernetes-ingress-default-cert
Namespace: testspace2
Labels: app.kubernetes.io/instance=testissue216
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kubernetes-ingress
app.kubernetes.io/version=1.10.10
helm.sh/chart=kubernetes-ingress-1.36.0
Annotations: helm.sh/hook: pre-install
helm.sh/hook-delete-policy: before-hook-creation
Type: kubernetes.io/tls
Data
====
tls.crt: 1164 bytes
tls.key: 1679 bytes
$ kubectl logs testissue216-kubernetes-ingress-b556d758d-pfd4g --namespace testspace2| grep ssl
2024/01/17 12:33:09 Default ssl certificate: testspace2/testissue216-kubernetes-ingress-default-cert
$ kubectl exec -it testissue216-kubernetes-ingress-b556d758d-pfd4g --namespace testspace2 -- ls -al /etc/haproxy/certs/frontend/0_testspace2_testissue216-kubernetes-ingress-default-cert.pem
-rw-r--r-- 1 haproxy haproxy 2843 Jan 17 12:33 /etc/haproxy/certs/frontend/0_testspace2_testissue216-kubernetes-ingress-default-cert.pem
Closing the ticket as no-op, nothing to fix here.
Ok. I'll go deep on this, probably misconfiguration or issue on ArgoCD part creating secret resource .yaml from parameters.
@Roxyrob We have been doing quite a few ArgoCD-related workarounds lately (https://github.com/haproxytech/helm-charts/issues/214, https://github.com/haproxytech/helm-charts/issues/211 and possibly more), so it is quite possible.
This issue is related to this ArgoCD behavior/bug interpreting Helm Hooks:
Argo hook not running on auto sync only on manual sync #9830
As explained here:
"automated sync doesn't trigger synchooks when it's a self-heal action"
As soon as ArgoCD will correct to look at "initial deployment" as "self-heal action", Job creating CRDs and Self-Signed certificate (Jobs annotated with helm hooks) will be run correctly in PreSync and GitOps with ArgoCD with syncPolicy.automated will work correctly.
This is a common issue for every deployment using Helm Hooks for environment preparation and actually the WORKAROUND is to start a Manual Sync after deplopyment (this can be annoying as haproxy in internal version to reach ArgoCD WebUI will not be immediately ready after deployment - it is needed to temporary use kubectl port-forward or argocd app sync cli).
Let us keep this issue open for now and I'll keep tracking https://github.com/argoproj/argo-cd/issues/9830.
ref: Default certificate does not exist #601
On haproxy ingress controller v1.10.10 I get this error in logs:
ERROR global.go:272 default certificate: annotation 'ssl-certificate': secret 'namespace/namespace-default-cert' does not exist namespace is custom namespace (not default).
I cannot use this ingress controller in https. If I manually create requested certificate that controller will remove it.
Sometimes ingress controller seems to deploy and auto create default certificates, with the same ingress configuration.
I know I can create custom one, but I need to know how haproxy ingress controller check and choose to create or not its default certificate to support tls without custom one ?