Request help: how to enable a private certification authority in gateway?

[MirtoBusico]

Hi all, I'm trying to setup the authz-keycloak plugin. My keycloak server replays at https://k6k.m01.net and it have key and certificate signed by my pricate Certification Authority. Trying to access the keycloak server from apisix pod I receve a "unable to get local issuer certificate" error

bash-5.1# curl -v https://k6k.m01.net
*   Trying 192.168.102.120:443...
* Connected to k6k.m01.net (192.168.102.120) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
bash-5.1# 

If I understand correctly, I have to setup the CA certificate in the tsl section of gateway

  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: ""
    certCAFilename: ""
    http2:
      enabled: true

Questions: 1) the certificate have to be inserted in existingCASecret or in certCAFilename? (in the last case where the file must reside respect the helm chart?)

2) the existingCASecret string requires a particular formatting? Is the example below valid?

  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: ""
    certCAFilename: "-----BEGIN CERTIFICATE-----
MIIEDTCCAvWgAwIBAgIUfUAyqeAGoxCGB6V/5qxOS/ZczrEwDQYJKoZIhvcNAQEL
BQAwgZUxCzAJBgNVBAYTAklUMQ4wDAYDVQQIDAVJdGFseTENMAsGA1UEBwwEUm9t
ZTEVMBMGA1UECgwMQnVzaWNvIE1pcnRvMRMwEQYDVQQLDApMYWJvcmF0b3J5MRUw
EwYDVQQDDAxCdXNpY28gTWlydG8xJDAiBgkqhkiG9w0BCQEWFW1pcnRvYnVzaWNv
QGdtYWlsLmNvbTAeFw0yMjAxMTcxNzQ2MDZaFw0zMjAxMTUxNzQ2MDZaMIGVMQsw
CQYDVQQGEwJJVDEOMAwGA1UECAwFSXRhbHkxDTALBgNVBAcMBFJvbWUxFTATBgNV
BAoMDEJ1c2ljbyBNaXJ0bzETMBEGA1UECwwKTGFib3JhdG9yeTEVMBMGA1UEAwwM
QnVzaWNvIE1pcnRvMSQwIgYJKoZIhvcNAQkBFhVtaXJ0b2J1c2ljb0BnbWFpbC5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC7T89OolDaC7YprD0l
3q20y0cchShqovk8Nzo12prDX8CCGxv6zexaHYVKk6qFfSGJJcgHcuxLIHvnflVH
Ugx9/LWOxaVz6N0i7z8hjnzxyMb2CXaTOpsbp0CdLeoEHjoZlESzAg6blIL4szPn
O2VpWfA10qTGWLC0z/QvcPENOVw3NLBZNJWbCl+WmcsCi+ttyzmSdMDS2ANnk6z0
mTqnFVtiCiRYQCXq0A5dr//Jva2q/h0sGoKHSV9Yt/StMB79uRVCGSpiCJAhalh1
8Zs+O9CPnkk+E2jnKn4jgI8FAY0Cii61nn15+/6BrHMu9h6/SO4oKj8i9UXPyodf
NW+3AgMBAAGjUzBRMB0GA1UdDgQWBBSCmgdUoJ6HXR5wMoX47koWO5HNnzAfBgNV
HSMEGDAWgBSCmgdUoJ6HXR5wMoX47koWO5HNnzAPBgNVHRMBAf8EBTADAQH/MA0G
CSqGSIb3DQEBCwUAA4IBAQBgZsZV87/e/8YauGLLGAen857V+NNvl1fMNPAF58O/
NG+iepahWxBJ5miEyMA6BH8ARUa1Q1fah8HC+/Q1dXEj17+h6d4QFS6PWBKp0a2N
MSnq0L4FYMnrUrhYxxyt4buNXDuYvaDit7lchKeHBJLBu/NBXH8WhMo/9g0Fg7YD
NRv6xg7wvYJf7YIc3RIg5bjklXKpdcvCZjuF8KVqv70x4eQx2m2zcf4CibvZKDFG
g/HY3btrW7fvhz9Ytj5w+SoCVLe3OKR0+koIyoGqsmiej9U4dbPTqVdsl3+XyfUF
oTpClYSDqa/kfmlT1o9FXpScRTQMOuHBiMYvEFiDBUGY
-----END CERTIFICATE-----"
    http2:
      enabled: true

[MirtoBusico]

Well reinstalling apisix with the tls chart modifications

  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: "-----BEGIN CERTIFICATE-----
MIIEDTCCAvWgAwIBAgIUfUAyqeAGoxCGB6V/5qxOS/ZczrEwDQYJKoZIhvcNAQEL
BQAwgZUxCzAJBgNVBAYTAklUMQ4wDAYDVQQIDAVJdGFseTENMAsGA1UEBwwEUm9t
ZTEVMBMGA1UECgwMQnVzaWNvIE1pcnRvMRMwEQYDVQQLDApMYWJvcmF0b3J5MRUw
EwYDVQQDDAxCdXNpY28gTWlydG8xJDAiBgkqhkiG9w0BCQEWFW1pcnRvYnVzaWNv
QGdtYWlsLmNvbTAeFw0yMjAxMTcxNzQ2MDZaFw0zMjAxMTUxNzQ2MDZaMIGVMQsw
CQYDVQQGEwJJVDEOMAwGA1UECAwFSXRhbHkxDTALBgNVBAcMBFJvbWUxFTATBgNV
BAoMDEJ1c2ljbyBNaXJ0bzETMBEGA1UECwwKTGFib3JhdG9yeTEVMBMGA1UEAwwM
QnVzaWNvIE1pcnRvMSQwIgYJKoZIhvcNAQkBFhVtaXJ0b2J1c2ljb0BnbWFpbC5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC7T89OolDaC7YprD0l
3q20y0cchShqovk8Nzo12prDX8CCGxv6zexaHYVKk6qFfSGJJcgHcuxLIHvnflVH
Ugx9/LWOxaVz6N0i7z8hjnzxyMb2CXaTOpsbp0CdLeoEHjoZlESzAg6blIL4szPn
O2VpWfA10qTGWLC0z/QvcPENOVw3NLBZNJWbCl+WmcsCi+ttyzmSdMDS2ANnk6z0
mTqnFVtiCiRYQCXq0A5dr//Jva2q/h0sGoKHSV9Yt/StMB79uRVCGSpiCJAhalh1
8Zs+O9CPnkk+E2jnKn4jgI8FAY0Cii61nn15+/6BrHMu9h6/SO4oKj8i9UXPyodf
NW+3AgMBAAGjUzBRMB0GA1UdDgQWBBSCmgdUoJ6HXR5wMoX47koWO5HNnzAfBgNV
HSMEGDAWgBSCmgdUoJ6HXR5wMoX47koWO5HNnzAPBgNVHRMBAf8EBTADAQH/MA0G
CSqGSIb3DQEBCwUAA4IBAQBgZsZV87/e/8YauGLLGAen857V+NNvl1fMNPAF58O/
NG+iepahWxBJ5miEyMA6BH8ARUa1Q1fah8HC+/Q1dXEj17+h6d4QFS6PWBKp0a2N
MSnq0L4FYMnrUrhYxxyt4buNXDuYvaDit7lchKeHBJLBu/NBXH8WhMo/9g0Fg7YD
NRv6xg7wvYJf7YIc3RIg5bjklXKpdcvCZjuF8KVqv70x4eQx2m2zcf4CibvZKDFG
g/HY3btrW7fvhz9Ytj5w+SoCVLe3OKR0+koIyoGqsmiej9U4dbPTqVdsl3+XyfUF
oTpClYSDqa/kfmlT1o9FXpScRTQMOuHBiMYvEFiDBUGY
-----END CERTIFICATE-----"
    certCAFilename: ""
    http2:
      enabled: true

The apisix pods never starts complaining it is not able to start (seems conflicting with istio CA)

Seems I don't understand how this works

[tokers]

@MirtoBusico You CA certificate should be saved in the Kubernetes Secret, and the caCerficiateFilename is used to indicate the key name in this Secret.

[MirtoBusico]

Thanks @tokers How can I do this? Which type of kubernetes secret? Is the caCerficiateFilename the secret name?

There is any example?

[tokers]

Thanks @tokers How can I do this? Which type of kubernetes secret? Is the caCerficiateFilename the secret name?

There is any example?

See https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix/README.md#gateway-parameters.

@MirtoBusico You CA certificate should be saved in the Kubernetes Secret, and the caCerficiateFilename is used to indicate the key name in this Secret.

See the above reply for learning the use of caCerficiateFilename.

And Kubernetes secret type doesn't matter.

[MirtoBusico]

@tokers Well maybe I don't understand the documentation

I created a generic secret using these commands:

sysop@m01serv:~/m01certs$ cd ~/m01certs
sysop@m01serv:~/m01certs$ ls -lh m01ca.*
-rw-r--r-- 1 root root 1,7K gen 17 18:43 m01ca.key
-rw-r--r-- 1 root root 1,5K gen 17 18:46 m01ca.pem
-rw-r--r-- 1 root root   41 feb 11 14:01 m01ca.srl
sysop@m01serv:~/m01certs$ kubectl -n kube-system create secret generic m01cacert --from-file=cert=./m01ca.pem
secret/m01cacert created
sysop@m01serv:~/m01certs$ kubectl describe secret m01cacert -n kube-system
Name:         m01cacert
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
cert:  1464 bytes
sysop@m01serv:~/m01certs$ 

Then I modified the values.yaml file in the apisix helm chart:

  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: "m01cacert"
    certCAFilename: "cert"
    http2:
      enabled: true

Then I installed apisix with:

kubectl create ns apisix
kubectl label namespace apisix istio-injection=enabled
helm install apisix apisix/apisix -f apisix-values.yaml \
--set ingress-controller.config.apisix.serviceNamespace=apisix \
--set ingress-controller.config.apisix.serviceName=apisix-admin \
--namespace apisix

Now the apisix pod never starts saying

MountVolume.SetUp failed for volume "ssl" : secret "m01cacert" not found

What I'm doing wrong?

[MirtoBusico]

Hi @tokers the first probem is that the secret is namespaced. So to have a valid secret it have to be defined in the apisix namespace

sysop@m01serv:~/m01certs$ kubectl -n apisix create secret generic m01cacert --from-file=cert=./m01ca.pem
secret/m01cacert created
sysop@m01serv:~/m01certs$ kubectl describe secret m01cacert -n apisix
Name:         m01cacert
Namespace:    apisix
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
cert:  1464 bytes
sysop@m01serv:~/m01certs$ 

Now apisix correctly start; but the "SSL certificate problem: unable to get local issuer certificate" error is still there. Trying to curl the keycloak server from apisix pod gives:

bash-5.1# curl -v https://k6k.m01.net
*   Trying 192.168.102.120:443...
* Connected to k6k.m01.net (192.168.102.120) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
bash-5.1# 

Seems that the secret is not correct or the secret key have to use a particular name.

[tokers]

@MirtoBusico The error reported here is self-explanation:

* TLSv1.3 (OUT), TLS alert, unknown CA (560):

Check out whether your certificate is self-signed, signed by some private CA. If so, specifying its CA certificate in the curl utility to make the certificate pass the verification.

[MirtoBusico]

Hi @tokers you are right As I said in previos comments I created my own CA and signed my certificates with this CA. My working configuration is obtained doing:

I create a sectre with my ownn CA certificate

sysop@m01serv:~/m01certs$ cd ~/m01certs
sysop@m01serv:~/m01certs$ ls -lh m01ca.*
-rw-r--r-- 1 root root 1,7K gen 17 18:43 m01ca.key
-rw-r--r-- 1 root root 1,5K gen 17 18:46 m01ca.pem
-rw-r--r-- 1 root root   41 feb 11 14:01 m01ca.srl
sysop@m01serv:~/m01certs$ kubectl -n apisix create secret generic m01cacert --from-file=cert=./m01ca.pem
secret/m01cacert created
sysop@m01serv:~/m01certs$ kubectl describe secret m01cacert -n apisix
Name:         m01cacert
Namespace:    apisix
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
cert:  1464 bytes
sysop@m01serv:~/m01certs$ 

Then I put these lines in apisix chart values.yaml

  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: "m01cacert"
    certCAFilename: "cert"
    http2:
      enabled: true

and installed apisix with

kubectl create ns apisix
kubectl label namespace apisix istio-injection=enabled
helm install apisix apisix/apisix -f apisix-values.yaml \
--set ingress-controller.config.apisix.serviceNamespace=apisix \
--set ingress-controller.config.apisix.serviceName=apisix-admin \
--namespace apisix

Strangely the curl command issued from the apisix pod still fails with the previous error but the apisix openid-connect plugin that accesses the url used in the curl command works correctly

I don't know is this is the correct behaviour

The curl command and his error

bash-5.1# curl -v https://k6k.m01.net/auth/realms/apisix_test_realm/.well-known/openid-configuration
*   Trying 192.168.102.120:443...
* Connected to k6k.m01.net (192.168.102.120) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
bash-5.1# 

The working apisix openid-connect plugin used in a route

{
    "client_id":"apisix",
    "client_secret":"CFejdjaiPNgGXMQub467j10OzcuK43tB",
    "discovery":"https://k6k.m01.net/auth/realms/apisix_test_realm/.well-known/openid-configuration",
    "scope":"openid profile",
    "bearer_only":false,
    "realm":"apisix_test_realm",
    "introspection_endpoint_auth_method":"client_secret_post",
    "redirect_uri":"https://www.m01.net/*",
    "access_token_in_authorization_header":true,
    "logout_path":"/logout"
}

Also I don't understand why the same url is refused if issued from the apisix pod and is accepted if used in the openid-connect plugin

[tokers]

@MirtoBusico I think the openid-connect plugin doesn't verify the peer's certificate strictly. And in the curl case, you can add the --cacert option to specify the CA certificate file and see what happens.

[MirtoBusico]

@tokers Ok. So it seems this is the correct behaviour. For me this request can be closed.

I'm just curious to understand the use for existingCASecret and certCAFilename in the tls section

  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: "m01cacert"
    certCAFilename: "cert"
    http2:
      enabled: true

[tokers]

@MirtoBusico existingCASecret is the name of Kubernetes Secret which contains the CA certificate, so basically you should create such Secret in advance; And certCAFilename is the KEY inside such Secret where the value is the CA certificate content.

[MirtoBusico]

@tokers yes. And this is exactly what I've done (see my previous replies). The question is why the curl from the apisix pod says unknown CA if the CA cert is loaded in apisix configuration?

[MirtoBusico]

Il 25/02/22 13:33, Alex Zhang ha scritto:

@tokers <https://github.com/tokers> yes. And this is exactly what
I've done (see my previous replies).

The question is why the curl from the apisix pod says unknown CA
if the CA cert is loaded in apisix configuration?

Just like you said, it's just loaded by apisix, then what does that have to do with curl?

Clearly I don't understand the use case for this certificate.

If I can load a CA certificate in apisix tls section, when and how I can use it to validate https requests?

— Reply to this email directly, view it on GitHub https://github.com/apache/apisix-helm-chart/issues/235#issuecomment-1050815232, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUTX5TYYVRQQA6ZJDYC3B3U45ZINANCNFSM5OIXL7XA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[tokers]

@tokers yes. And this is exactly what I've done (see my previous replies). The question is why the curl from the apisix pod says unknown CA if the CA cert is loaded in apisix configuration?

I checked the case you pasted, you login to the APISIX pod but sent requests to k6k.m01.net , I don't know how you handle the DNS resolving, but you should make sure requests were sent to the APISIX cluster.

[MirtoBusico]

Well my framework is:

• a client machine for browser access - resolves addresses with these lines in /etc/hosts file

  
  192.168.102.120  m01serv m01serv.m01.net
  192.168.102.120  k6k k6k.m01.net
  192.168.102.120  njsapp njsapp.m01.net

192.168.102.121 reg.m01.net 192.168.102.121 m01km m01km.m01.net 192.168.102.121 www.m01.net 192.168.102.121 api.m01.net 192.168.102.121 www2.m01.net 192.168.102.121 api2.m01.net 192.168.102.121 lh.m01.net

192.168.102.122 m01kw1 m01kw1.m01.net

192.168.102.123 m01kw2 m01kw2.m01.net

-  a virtual machine (m01serv) for cluster external services (as keycloak, DNS ...) resolves addresses using the local DNS service
- a kubernetes cluster that uses K3S - every node resolves addresses using the DNS service on m01serv
  - m01km master and worker node 
  - m01kw1 a worker node
  - m01kw2 a worker node

In this framework the Istio service mesh is installed on every node and Apisix is installed with loadbalancer access as ingress controller

In all the nodes the private CA certificate is installed in the OS, so you can access the keycloak server (https://k6k.m01.net) without having the "unable to get local issuer certificate" issue

With apisix used as ingress controller for the cluster I'm using the **openid-connect** and **authz-keycloak** plugins and they works correctly accessing  the keycloak server (https://k6k.m01.net) that is outside the cluster.
Maybe those plugins don't do a strict certificate verification

In my mind when the plugin accesses the external keycloak server the request should be originated from the apisix pod and I should be able to curl the keycloak server from the same pod.

BTW until the two plugins work correctly, i don't need to have the private CA certificate recognized by apisix
You can close the thread.
Thanks for your time

[tokers]

OK, so k6k.m01.net is the keycloak serivce address, you sent requests to keycloak and find the CA problems. And it seems that your keycloak service is not proxied by APISIX since you said:

With apisix used as ingress controller for the cluster I'm using the openid-connect and authz-keycloak plugins and they works correctly accessing the keycloak server (https://k6k.m01.net/) that is outside the cluster. Maybe those plugins don't do a strict certificate verification

So your last question:

The question is why the curl from the apisix pod says unknown CA if the CA cert is loaded in apisix configuration?

You don't send requests to APISIX at all, It's not a question, APISIX doesn't handle your requests. And I already said:

I checked the case you pasted, you login to the APISIX pod but sent requests to k6k.m01.net , I don't know how you handle the DNS resolving, but you should make sure requests were sent to the APISIX cluster.