k8s-ci-robot
commented
3 years ago @voigt: The label(s) triage/support cannot be applied, because the repository doesn't have them
In response to [this](https://github.com/kubernetes/ingress-nginx/issues/6876):
>
>**NGINX Ingress controller version**: `0.40.1`
>
>**Kubernetes version** (use `kubectl version`): `1.16.4`
>
>**Environment**:
>
>- **Cloud provider or hardware configuration**: AWS EC2 c5n.large
>- **OS** (e.g. from /etc/os-release): Ubuntu 18.04.5 LTS
>- **Kernel** (e.g. `uname -a`): 5.4.0-1037-aws
>- **Install tools**: -
>- **Others**: cert-manager v0.9.1
>
>**What happened**:
>
>Hi,
>we just recently updated our ingress controllers from `0.24.1` to `0.40.1`. So far so good.
>
>What we now noticed is, that our intermediate certificates are not any longer part of the ssl-chain on an ssl connect:
>
>, CN = service.domain.com
>verify error:num=20:unable to get local issuer certificate
>verify return:1
>depth=0 C = IE, ST = City, L = Country, O = Organisation, OU = , CN = service.domain.com
>verify error:num=21:unable to verify the first certificate
>verify return:1
>---
>Certificate chain
> 0 s:C = IE, ST = City, L = Country, O = Organisation, OU = , CN = service.domain.com
> i:DC = com, DC = , DC = ORGPKI, CN = , O = Organisation
>---
>Server certificate
>-----BEGIN CERTIFICATE-----
>
>-----END CERTIFICATE-----
>[...]
>---
>closed
>```
>
openssl s_client -connect on a 0.40.1 endpoint
> >```bash >$ openssl s_client -connect service.domain.com:443 -servername service.domain.com >CONNECTED(00000005) >depth=0 C = IE, ST = City, L = Country, O = Organisation, OU =What you expected to happen:
An output, similar to environments where we are running 0.24.1 (without)
openssl s_client -connect on a 0.24.1 endpoint
``` $ openssl s_client -connect otherservice.domain.com:443 -servername otherservice.domain.com CONNECTED(00000005) depth=2 DC = com, DC =, DC = ORGPKI, CN = -05 verify error:num=19:self signed certificate in certificate chain --- Certificate chain 0 s:C = IE, ST = , L = , O = , OU = , CN = otherservice.domain.com i:DC = com, DC = , DC = ORGPKI, CN = -01, O = 1 s:DC = com, DC = , DC = ORGPKI, CN = -01, O = i:DC = com, DC = , DC = ORGPKI, CN = -05 2 s:DC = com, DC = , DC = ORGPKI, CN = -05 i:DC = com, DC = , DC = ORGPKI, CN = -05 3 s:DC = com, DC = , DC = ORGPKI, CN = -01, O = i:DC = com, DC = , DC = ORGPKI, CN = -05 --- [...] ``` How to reproduce it:
$ openssl s_client -connect service.domain.com:443 -servername service.domain.comAnything else we need to know:
We have multiple environments running and only those with the updated version IC
0.40.1show this behavior.from the Helm-Chart we enabled the following
extraArgs. Please note, that with 0.24.1 we also enabledenable-dynamic-certificates: "false", which does not seem to exist anymore. I'm explicitly pointing that out, as I found issue #4245.enable-ssl-passthrough: "true" enable-ssl-chain-completion: "true"
- Arguments with which the IC is started:
all nginx-ingress-controller arguments used
``` spec: containers: - args: - /nginx-ingress-controller - --default-backend-service=ingress/ingress-nginx-defaultbackend - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller - --election-id=ingress-controller-leader - --ingress-class=portal - --configmap=ingress/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key - --enable-ssl-chain-completion=true - --enable-ssl-passthrough=true ```
- A sample ingress (redacted) - we make use of
nginx.ingress.kubernetes.io/configuration-snippet, which hasn't changed after we updated to 0.40.1:Sample Ingress
```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: ingress.kubernetes.io/force-ssl-redirect: "true" ingress.kubernetes.io/rewrite-target: / kubernetes.io/ingress.class: portal nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false" nginx.ingress.kubernetes.io/auth-tls-secret: default/casecret nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1" nginx.ingress.kubernetes.io/configuration-snippet: | set $allowed 0; if ($ssl_client_i_dn = "CN=,DC= ,DC= ,DC= ") { set $allowed 1; } if ($ssl_client_i_dn = "CN= ,DC= ,DC= ,DC= ") { set $allowed 1; } if ($ssl_client_i_dn = "CN= ,DC= ,DC= ,DC= ") { set $allowed 1; } if ($ssl_client_s_dn = "C= ,O= ,OU= ,CN= ") { set $allowed 1; } if ($allowed != 1) { return 401 $ssl_client_s_dn; } labels: app: service name: service-ingress namespace: default spec: rules: - host: service.domain.com http: paths: - backend: serviceName: service servicePort: 8080 path: / tls: - hosts: - service.domain.com secretName: service.domain.com status: loadBalancer: ingress: - ip: 100.69.155.90 ```
- We use cert-manager
v0.9.1(I know, pretty old - the upgrade is WIP) to get TLS certificates that are then inserted into our ingress resources. So for the sake of completeness a sample Kubernetes secret as it is used with 0.24.1 and 0.40.1:kubectl get secret service.domain.com -oyaml
```yaml root@4de00908bc18:/# kubectl get secret service.domain.com -oyaml apiVersion: v1 data: ca.crt:tls.crt: kind: Secret metadata: annotations: certmanager.k8s.io/alt-names: service.domain.com certmanager.k8s.io/common-name: service.domain.com certmanager.k8s.io/ip-sans: "" certmanager.k8s.io/issuer-kind: ClusterIssuer certmanager.k8s.io/issuer-name: vault-issuer creationTimestamp: "2019-06-21T08:42:00Z" labels: certmanager.k8s.io/certificate-name: service.domain.com name: service.domain.com namespace: default type: kubernetes.io/tls ```
- Note that TLS secrets, as well as Ingress resources, did not change after the upgrade
I'm a bit puzzled here...
- Why does
--enable-dynamic-certificates=falsenot exist anymore in 0.40.1, and can it be related? (#4245)- Is it possible, that the requirements for the certificate format changed with a later version of the IC? Does cert-manager create secrets that the newer IC versions can't deal with?
- And most importantly: how do I get back the full ssl-chain? 😅
- I already tried my luck in the slack channel, so far without any response, that's why I try it here...
Any help kindly appreciated!
/triage support
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
voigt
fejta-bot
artemkrim
muffl0n
k8s-triage-robot
Timoses
tschechniker
gmarkey
jtermine
NGINX Ingress controller version:
0.40.1Kubernetes version (use
kubectl version):1.16.4Environment:
uname -a): 5.4.0-1037-awsWhat happened:
Hi, we just recently updated our ingress controllers from
0.24.1to0.40.1. So far so good.What we now noticed is, that our intermediate certificates are not any longer part of the ssl-chain on an ssl connect:
openssl s_client -connect on a 0.40.1 endpoint
```bash $ openssl s_client -connect service.domain.com:443 -servername service.domain.com CONNECTED(00000005) depth=0 C = IE, ST = City, L = Country, O = Organisation, OU =What you expected to happen:
An output, similar to environments where we are running 0.24.1 (without)
openssl s_client -connect on a 0.24.1 endpoint
``` $ openssl s_client -connect otherservice.domain.com:443 -servername otherservice.domain.com CONNECTED(00000005) depth=2 DC = com, DC =How to reproduce it:
Anything else we need to know:
We have multiple environments running and only those with the updated version IC
0.40.1show this behavior.from the Helm-Chart we enabled the following
extraArgs. Please note, that with 0.24.1 we also enabledenable-dynamic-certificates: "false", which does not seem to exist anymore. I'm explicitly pointing that out, as I found issue #4245.all nginx-ingress-controller arguments used
``` spec: containers: - args: - /nginx-ingress-controller - --default-backend-service=ingress/ingress-nginx-defaultbackend - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller - --election-id=ingress-controller-leader - --ingress-class=portal - --configmap=ingress/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key - --enable-ssl-chain-completion=true - --enable-ssl-passthrough=true ```nginx.ingress.kubernetes.io/configuration-snippet, which hasn't changed after we updated to 0.40.1:Sample Ingress
```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: ingress.kubernetes.io/force-ssl-redirect: "true" ingress.kubernetes.io/rewrite-target: / kubernetes.io/ingress.class: portal nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false" nginx.ingress.kubernetes.io/auth-tls-secret: default/casecret nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1" nginx.ingress.kubernetes.io/configuration-snippet: | set $allowed 0; if ($ssl_client_i_dn = "CN=v0.9.1(I know, pretty old - the upgrade is WIP) to get TLS certificates that are then inserted into our ingress resources. So for the sake of completeness a sample Kubernetes secret as it is used with 0.24.1 and 0.40.1:kubectl get secret service.domain.com -oyaml
```yaml root@4de00908bc18:/# kubectl get secret service.domain.com -oyaml apiVersion: v1 data: ca.crt:I'm a bit puzzled here...
--enable-dynamic-certificates=falsenot exist anymore in 0.40.1, and can it be related? (#4245)Any help kindly appreciated!
/triage support