How to set a private CA certificate in istiod?

Cannot use user authentication

[ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ x] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ ] Upgrade

Expected behavior

istiod can use a private CA to access with https an externale Keycloak server for user authentication

Steps to reproduce the bug

install istio 1.8.0 on a K3S cluster
create a private CA and generate key/certificate for keycloak
install the private CA certificate on every cluster node
create a keycloak server accessible through https (eg https://k6k2.h.net)
create a RequestAuthentication for istioingressgateway pointing to the keycloak server

The RequestAuthentication doesn't work because istiod refuses the certificate Istiod do not accept the certificate

sysop@h2km:~$ kubectl exec -it "$(kubectl get pod -l app=istiod -n istio-system -o jsonpath='{.items[0].metadata.name}')" -n istio-system -- curl  https://k6k2.h.net/auth/realms/h2project
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
command terminated with exit code 60

The certificate is accepted on master node

sysop@h2km:~$ curl  https://k6k2.h.net/auth/realms/h2project
{"realm":"h2project","public_key":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoXta3avrLBv+WEsYBtCzGcYIM+n5qNlj35hWT4zQU6aetSAG8vDNVxOePrRazFhBI97SLRfuIG0Jye6vH7Dwr81mdLROebnF8BmrIJ5gIfyIiDjsZ77YtMg1eUtZeVwxwMTEIEGDIW3jFgPeA5ICLkdbfdMc49OUHbz4WDjKtX2yQDqtzJtX58PhWXPwBT2tq6mqYw6jz6ZKMCIdtgFX5C1Dc/gJbCbpAf4EO6pvdJti2bwMubkHtn3jWaPdMhlyRQKKTBFDt2IVfVqPiCapWajIZixh+/1S1bfsNO5jAEeDbVAXGrXIC3YRgcjj/jfAtqu6OJ4vuERktDgFmPmGGQIDAQAB","token-service":"https://k6k2.h.net/auth/realms/h2project/protocol/openid-connect","account-service":"https://k6k2.h.net/auth/realms/h2project/account","tokens-not-before":0}

How can I install the private CA certificate in istiod? There is any other method to create a RequestAuthentication using https?

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

sysop@h2dev:~$ istioctl version --remote
client version: 1.8.0
control plane version: 1.8.0
data plane version: 1.8.0 (13 proxies)
sysop@h2dev:~$ kubectl version --short
Client Version: v1.19.4
Server Version: v1.19.3+k3s3
sysop@h2dev:~$

How was Istio installed?

istioctl install --set profile=demo -y

Environment where the bug was observed (cloud vendor, OS, etc)

3 KVM virtual machines with Ubuntu 20.04 K3S master and two workers on the 3 virtual machines

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.

Tried again using CA key and certificate from the isio distribution Taken from [where you downloaded istio]/istio-1.8.0/samples/certs files: ca-cert.pem ca-key.pem

Then I setup a ketcloak server using certificates generated by this CA

Nothing changes istiod complains that there is an unknown authority

Istiod log shows

2020-12-10T16:52:36.874171Z info    ads RDS: PUSH for node:httpbin-74fb669cc6-vd2bm.default resources:20
2020-12-10T16:52:36.951748Z error   model   Failed to fetch public key from "https://keyk.h.net/auth/realms/h2project/protocol/openid-connect/certs": Get "https://keyk.h.net/auth/realms/h2project/protocol/openid-connect/certs": x509: certificate signed by unknown authority
2020-12-10T16:52:36.951786Z error   Failed to fetch jwt public key from "https://keyk.h.net/auth/realms/h2project/protocol/openid-connect/certs": Get "https://keyk.h.net/auth/realms/h2project/protocol/openid-connect/certs": x509: certificate signed by unknown authority
2020-12-10T16:52:36.966766Z error   model   Failed to fetch public key from "https://keyk.h.net/auth/realms/h2project/protocol/openid-connect/certs": Get "https://keyk.h.net/auth/realms/h2project/protocol/openid-connect/certs": x509: certificate signed by unknown authority
2020-12-10T16:52:36.966792Z error   Failed to fetch jwt public key from "https://keyk.h.net/auth/realms/h2project/protocol/openid-connect/certs": Get "https://keyk.h.net/auth/realms/h2project/protocol/openid-connect/certs": x509: certificate signed by unknown authority

A curl from the istiod pod gives the same error

sysop@h2dev:~$ kubectl exec -it "$(kubectl get pod -l app=istiod -n istio-system -o jsonpath='{.items[0].metadata.name}')" -n istio-system -- curl  https://keyk.h.net/auth/realms/h2project
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
command terminated with exit code 60
sysop@h2dev:~$

The node where istiod runs is h2kw2

sysop@h2dev:~$ kubectl get pod -n istio-system -o wide
NAME                                    READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
istiod-5c6d68885d-d6bd9                 1/1     Running   10         16d   10.42.2.89    h2kw2   <none>           <none>
kiali-7476977cf9-x55cf                  1/1     Running   8          15d   10.42.1.109   h2kw1   <none>           <none>

The same curl on the istiod pod node works correctly

sysop@h2kw2:~$ curl  https://keyk.h.net/auth/realms/h2project
{"realm":"h2project","public_key":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoXta3avrLBv+WEsYBtCzGcYIM+n5qNlj35hWT4zQU6aetSAG8vDNVxOePrRazFhBI97SLRfuIG0Jye6vH7Dwr81mdLROebnF8BmrIJ5gIfyIiDjsZ77YtMg1eUtZeVwxwMTEIEGDIW3jFgPeA5ICLkdbfdMc49OUHbz4WDjKtX2yQDqtzJtX58PhWXPwBT2tq6mqYw6jz6ZKMCIdtgFX5C1Dc/gJbCbpAf4EO6pvdJti2bwMubkHtn3jWaPdMhlyRQKKTBFDt2IVfVqPiCapWajIZixh+/1S1bfsNO5jAEeDbVAXGrXIC3YRgcjj/jfAtqu6OJ4vuERktDgFmPmGGQIDAQAB","token-service":"https://keyk.h.net/auth/realms/h2project/protocol/openid-connect","account-service":"https://keyk.h.net/auth/realms/h2project/account","tokens-not-before":0}sysop@h2kw2:~$

Questions: Where istiod take the list of known certification authorities? How to add a new certification authority certificate to this list?

i have the same issue, please tell us how to do that

You can set the jwksResolverExtraRootCA to use an extra root CA for your jwks server, see https://discuss.istio.io/t/setting-pilot-jwksresolverextrarootca-in-istiooperator/9058/4

Meanwhile I think we should improve the documentation to make the solution a little bit more clear, @xulingqing could you take a look at this and update the doc:

We can add the solution in the above link of using jwksResolverExtraRootCA in the istio.io in https://istio.io/latest/docs/ops/configuration/security/
We should also mention this in the API in https://github.com/istio/api/blob/01ce8d4813147477582ae7de864d2f2723400408/security/v1beta1/jwt.proto#L93

You can set the jwksResolverExtraRootCA to use an extra root CA for your jwks server, see https://discuss.istio.io/t/setting-pilot-jwksresolverextrarootca-in-istiooperator/9058/4

Meanwhile I think we should improve the documentation to make the solution a little bit more clear, @xulingqing could you take a look at this and update the doc:
1. We can add the solution in the above link of using `jwksResolverExtraRootCA` in the istio.io in https://istio.io/latest/docs/ops/configuration/security/

2. We should also mention this in the API in https://github.com/istio/api/blob/01ce8d4813147477582ae7de864d2f2723400408/security/v1beta1/jwt.proto#L93

How can I change the istio operator? I can do this in a working istio installation os I have to do this before installing instio?

I tried Plug in CA Certificates then I used this CA to sign the keycloak server (running external to the cluster)

Sorry @yangminzhu I'm trying your suggestion but seems the jwksResolverExtraRootCA I created is ignored.

I istalled Istio 1.8.1 using

istioctl install --set profile=demo -y

Then I tried to create an update yaml to install jwksResolverExtraRootCA. The file contains:

sysop@hdev:~/software/hproject$ cat IstioOperator.yaml 
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: update-istio-operator
spec:
  meshConfig:
    accessLogEncoding: TEXT
    accessLogFile: "/dev/stdout"
    accessLogFormat: ""
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  values:
    pilot:
      jwksResolverExtraRootCA: |
        -----BEGIN CERTIFICATE-----
        MIIFFDCCAvygAwIBAgIUR5/JByU16/FqSqqhiZ1Lk5m0cDYwDQYJKoZIhvcNAQEL
        BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjAxMjIx
        MTEzNjI4WhcNMzAxMjE5MTEzNjI4WjAiMQ4wDAYDVQQKDAVJc3RpbzEQMA4GA1UE
        AwwHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAJ7peieP
        6beOn8815tWDwW7Ro7SMZzlOJxAYZFxwtqi9HsKYAOtMcbjYMf0bdWES6WxmXI3N
        NSbVtgZeQESG89ZL5l+CoCSPht5yBI7QnvbSO99qxxF8DawGlmSXBKTaReIro3Z8
        mBKFT9fNT+JK+ORZR5RI3juaqKElL3uRnSXTOT51SGjJqzVIJ+iD7cuBYTta1vXl
        eDwM2CtWW9hKYMbvEwhP6sEr1JlQsuPJKrqerQt/8uiH3oF5iZ8GND12iMlG8gzg
        et9DweUX5cR9AwrBON7+SfEoAnWObAcY19h+lDCDujWyHJMI5vPe09Ampoa5T7+q
        WtMM2Fjm4XvFEu4kwthqT0mxmtXvtlPlBTO/YLZ+qhVjAOS5K7HU47Bu51Q0hOky
        irxkU7kXZrvSekXWYFimytmjjN2nCAhbeXG8bmJmOxBcgDHMea1SFPyHQHIsQNRW
        91Hg2VNk9ooqYyMLxuYy4TdAYUg+t+TNLGsAYkY4bWrII9D/CkvBlMiFYerbS6ds
        4ee7n6jJNhmzJcg/ELoocvPBaDLjll6mBjCVboSycwRHN5Fd96Jc+AWhHUTxCv1z
        252Jql8cKR+YGok4xNC6qWR7qIPKCOpac7gRfcnD15eRW5ynhqy82tHQY7m9Ss5g
        4vcEHjnFLYDN7LN6Od9MmxTsxlFq7DdNzG7rAgMBAAGjQjBAMB0GA1UdDgQWBBSM
        wXc6WuZO7HtmIGaRpGd4l/uitTAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE
        AwIC5DANBgkqhkiG9w0BAQsFAAOCAgEAdSHQ6EG7VUMMcysDgcvxWPKWgbliFtWq
        qJ4xMg8CI3Ya96zcUHdvPayzkIde93KEyt6GUzUYOjuo4cmLswDr0Jxl2CMcEchY
        cLW9oW+4FkyhE4C7/6rGbDmYNUu2GyJJ6Ut9px+jGdAO9uOxmk1jsGD+fahkTi7H
        ZXpQC48iEublkUZHFK2jzsWk4920lWlYLCOzfqRQH+btXQ5Wlmy/yXfpqwqk9WQd
        AYw1frXhULaUp+LYiICfQjvYiK9/zNCrLM865MEZmGVmzMGjRhSJxNA9ZlGI/yEy
        1Arx51Ci3jSFMjJBpVGM5TGv0PUvGmbMJBp/oRiiQr/BxOkeITrvdGT/QyYOdWGT
        dEgBW9pW6wAurq7qUaaR3IKxdev2FsvF+1rWwENk1SPnyA2+wGGPLhAnDn6ED/uk
        1hBZWQiPIkCxwHmkRTXjAJceBtV/e44HSdubEsBrcVr+cd0BPFqqDwMz19vY4FYo
        /fcyUAwXuh4MUDQO680PxFW5sOvQ6c8rd7ybybj7aiwk8EO/fesZPfOrwYJtNsaY
        u5Qw/LxTmN9MAkr280FFMWFeN+WfcsElI7enTURLDfQbVBtKbdT5HwfoaixJrQqW
        6Hg4pCH9PgCy2RAaSNVUriRqwc8xSLvs/P8pWtGu/1zaocyvo9I9u0oJPDslrlWY
        SKCsExuPHWs=
        -----END CERTIFICATE-----

sysop@hdev:~/software/hproject$

If I look for IstioOperator I see

sysop@hdev:~/software/hproject$ kubectl get istiooperator --all-namespaces
NAMESPACE      NAME                    REVISION   STATUS   AGE
istio-system   installed-state                             33d
istio-system   update-istio-operator                       21h
sysop@hdev:~/software/hproject$

Clearly I'm doing something wrong or I miss something.

What can I do?

Sorry @yangminzhu I've done another try without luck

The step I followed: Dumped the running istio operator configuration using

kubectl -n istio-system get IstioOperator installed-state -o yaml > installed-state.yaml

Then I added the certificate after line 399

...
399   values:
400     pilot:
401       jwksResolverExtraRootCA: |
402         -----BEGIN CERTIFICATE-----
403         MIIFFDCCAvygAwIBAgIUR5/JByU16/FqSqqhiZ1Lk5m0cDYwDQYJKoZIhvcNAQEL
404         BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjAxMjIx
405         MTEzNjI4WhcNMzAxMjE5MTEzNjI4WjAiMQ4wDAYDVQQKDAVJc3RpbzEQMA4GA1UE
406         AwwHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAJ7peieP
407         6beOn8815tWDwW7Ro7SMZzlOJxAYZFxwtqi9HsKYAOtMcbjYMf0bdWES6WxmXI3N
408         NSbVtgZeQESG89ZL5l+CoCSPht5yBI7QnvbSO99qxxF8DawGlmSXBKTaReIro3Z8
409         mBKFT9fNT+JK+ORZR5RI3juaqKElL3uRnSXTOT51SGjJqzVIJ+iD7cuBYTta1vXl
410         eDwM2CtWW9hKYMbvEwhP6sEr1JlQsuPJKrqerQt/8uiH3oF5iZ8GND12iMlG8gzg
411         et9DweUX5cR9AwrBON7+SfEoAnWObAcY19h+lDCDujWyHJMI5vPe09Ampoa5T7+q
412         WtMM2Fjm4XvFEu4kwthqT0mxmtXvtlPlBTO/YLZ+qhVjAOS5K7HU47Bu51Q0hOky
413         irxkU7kXZrvSekXWYFimytmjjN2nCAhbeXG8bmJmOxBcgDHMea1SFPyHQHIsQNRW
414         91Hg2VNk9ooqYyMLxuYy4TdAYUg+t+TNLGsAYkY4bWrII9D/CkvBlMiFYerbS6ds
415         4ee7n6jJNhmzJcg/ELoocvPBaDLjll6mBjCVboSycwRHN5Fd96Jc+AWhHUTxCv1z
416         252Jql8cKR+YGok4xNC6qWR7qIPKCOpac7gRfcnD15eRW5ynhqy82tHQY7m9Ss5g
417         4vcEHjnFLYDN7LN6Od9MmxTsxlFq7DdNzG7rAgMBAAGjQjBAMB0GA1UdDgQWBBSM
418         wXc6WuZO7HtmIGaRpGd4l/uitTAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE
419         AwIC5DANBgkqhkiG9w0BAQsFAAOCAgEAdSHQ6EG7VUMMcysDgcvxWPKWgbliFtWq
420         qJ4xMg8CI3Ya96zcUHdvPayzkIde93KEyt6GUzUYOjuo4cmLswDr0Jxl2CMcEchY
421         cLW9oW+4FkyhE4C7/6rGbDmYNUu2GyJJ6Ut9px+jGdAO9uOxmk1jsGD+fahkTi7H
422         ZXpQC48iEublkUZHFK2jzsWk4920lWlYLCOzfqRQH+btXQ5Wlmy/yXfpqwqk9WQd
423         AYw1frXhULaUp+LYiICfQjvYiK9/zNCrLM865MEZmGVmzMGjRhSJxNA9ZlGI/yEy
424         1Arx51Ci3jSFMjJBpVGM5TGv0PUvGmbMJBp/oRiiQr/BxOkeITrvdGT/QyYOdWGT
425         dEgBW9pW6wAurq7qUaaR3IKxdev2FsvF+1rWwENk1SPnyA2+wGGPLhAnDn6ED/uk
426         1hBZWQiPIkCxwHmkRTXjAJceBtV/e44HSdubEsBrcVr+cd0BPFqqDwMz19vY4FYo
427         /fcyUAwXuh4MUDQO680PxFW5sOvQ6c8rd7ybybj7aiwk8EO/fesZPfOrwYJtNsaY
428         u5Qw/LxTmN9MAkr280FFMWFeN+WfcsElI7enTURLDfQbVBtKbdT5HwfoaixJrQqW
429         6Hg4pCH9PgCy2RAaSNVUriRqwc8xSLvs/P8pWtGu/1zaocyvo9I9u0oJPDslrlWY
430         SKCsExuPHWs=
431         -----END CERTIFICATE-----
432     base:
433       enableCRDTemplates: false
...

After I aplied the file with

kubectl apply -f installed-state.yaml --overwrite=true

But when I look at the new installed version the certificate was dropped

sysop@hdev:~/software/hproject$ kubectl -n istio-system get IstioOperator installed-state -o yaml > installed-state-after.yaml
sysop@hdev:~/software/hproject$ diff installed-state.yaml installed-state-after.yaml 
7c7
<       {"apiVersion":"install.istio.io/v1alpha1","kind":"IstioOperator","metadata":{"annotations":{"install.istio.io/ignoreReconcile":"true"},"creationTimestamp":null,"name":"installed-state","namespace":"istio-system"},"spec":{"addonComponents":{"istiocoredns":{"enabled":false}},"components":{"base":{"enabled":true},"cni":{"enabled":false},"egressGateways":[{"enabled":true,"k8s":{"env":[{"name":"ISTIO_META_ROUTER_MODE","value":"standard"}],"hpaSpec":{"maxReplicas":5,"metrics":[{"resource":{"name":"cpu","targetAverageUtilization":80},"type":"Resource"}],"minReplicas":1,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"istio-egressgateway"}},"resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"40Mi"}},"service":{"ports":[{"name":"http2","port":80,"protocol":"TCP","targetPort":8080},{"name":"https","port":443,"protocol":"TCP","targetPort":8443},{"name":"tls","port":15443,"protocol":"TCP","targetPort":15443}]},"strategy":{"rollingUpdate":{"maxSurge":"100%","maxUnavailable":"25%"}}},"name":"istio-egressgateway"}],"ingressGateways":[{"enabled":true,"k8s":{"env":[{"name":"ISTIO_META_ROUTER_MODE","value":"standard"}],"hpaSpec":{"maxReplicas":5,"metrics":[{"resource":{"name":"cpu","targetAverageUtilization":80},"type":"Resource"}],"minReplicas":1,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"istio-ingressgateway"}},"resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"40Mi"}},"service":{"ports":[{"name":"status-port","port":15021,"protocol":"TCP","targetPort":15021},{"name":"http2","port":80,"protocol":"TCP","targetPort":8080},{"name":"https","port":443,"protocol":"TCP","targetPort":8443},{"name":"tcp","port":31400,"protocol":"TCP","targetPort":31400},{"name":"tls","port":15443,"protocol":"TCP","targetPort":15443}]},"strategy":{"rollingUpdate":{"maxSurge":"100%","maxUnavailable":"25%"}}},"name":"istio-ingressgateway"}],"istiodRemote":{"enabled":false},"pilot":{"enabled":true,"k8s":{"env":[{"name":"PILOT_TRACE_SAMPLING","value":"100"}],"readinessProbe":{"httpGet":{"path":"/ready","port":8080},"initialDelaySeconds":1,"periodSeconds":3,"timeoutSeconds":5},"resources":{"requests":{"cpu":"10m","memory":"100Mi"}},"strategy":{"rollingUpdate":{"maxSurge":"100%","maxUnavailable":"25%"}}}}},"hub":"docker.io/istio","meshConfig":{"accessLogFile":"/dev/stdout","defaultConfig":{"proxyMetadata":{}},"enablePrometheusMerge":true},"profile":"demo","tag":"1.8.2","values":{"base":{"enableCRDTemplates":false,"validationURL":""},"clusterResources":true,"gateways":{"istio-egressgateway":{"autoscaleEnabled":false,"env":{},"name":"istio-egressgateway","secretVolumes":[{"mountPath":"/etc/istio/egressgateway-certs","name":"egressgateway-certs","secretName":"istio-egressgateway-certs"},{"mountPath":"/etc/istio/egressgateway-ca-certs","name":"egressgateway-ca-certs","secretName":"istio-egressgateway-ca-certs"}],"type":"ClusterIP","zvpn":{}},"istio-ingressgateway":{"autoscaleEnabled":false,"env":{},"name":"istio-ingressgateway","secretVolumes":[{"mountPath":"/etc/istio/ingressgateway-certs","name":"ingressgateway-certs","secretName":"istio-ingressgateway-certs"},{"mountPath":"/etc/istio/ingressgateway-ca-certs","name":"ingressgateway-ca-certs","secretName":"istio-ingressgateway-ca-certs"}],"type":"LoadBalancer","zvpn":{}}},"global":{"arch":{"amd64":2,"ppc64le":2,"s390x":2},"configValidation":true,"defaultNodeSelector":{},"defaultPodDisruptionBudget":{"enabled":true},"defaultResources":{"requests":{"cpu":"10m"}},"imagePullPolicy":"","imagePullSecrets":[],"istioNamespace":"istio-system","istiod":{"enableAnalysis":false},"jwtPolicy":"third-party-jwt","logAsJson":false,"logging":{"level":"default:info"},"meshExpansion":{"enabled":false,"useILB":false},"meshNetworks":{},"mountMtlsCerts":false,"multiCluster":{"clusterName":"","enabled":false},"network":"","omitSidecarInjectorConfigMap":false,"oneNamespace":false,"operatorManageWebhooks":false,"pilotCertProvider":"istiod","priorityClassName":"","proxy":{"autoInject":"enabled","clusterDomain":"cluster.local","componentLogLevel":"misc:error","enableCoreDump":false,"excludeIPRanges":"","excludeInboundPorts":"","excludeOutboundPorts":"","image":"proxyv2","includeIPRanges":"*","logLevel":"warning","privileged":false,"readinessFailureThreshold":30,"readinessInitialDelaySeconds":1,"readinessPeriodSeconds":2,"resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"40Mi"}},"statusPort":15020,"tracer":"zipkin"},"proxy_init":{"image":"proxyv2","resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"10Mi"}}},"sds":{"token":{"aud":"istio-ca"}},"sts":{"servicePort":0},"tracer":{"datadog":{},"lightstep":{},"stackdriver":{},"zipkin":{}},"useMCP":false},"istiocoredns":{"coreDNSImage":"coredns/coredns","coreDNSPluginImage":"istio/coredns-plugin:0.2-istio-1.1","coreDNSTag":"1.6.2"},"istiodRemote":{"injectionURL":""},"pilot":{"autoscaleEnabled":false,"autoscaleMax":5,"autoscaleMin":1,"configMap":true,"cpu":{"targetAverageUtilization":80},"deploymentLabels":null,"enableProtocolSniffingForInbound":true,"enableProtocolSniffingForOutbound":true,"env":{"ENABLE_LEGACY_FSGROUP_INJECTION":false},"image":"pilot","keepaliveMaxServerConnectionAge":"30m","nodeSelector":{},"replicaCount":1,"traceSampling":1},"sidecarInjectorWebhook":{"enableNamespacesByDefault":false,"objectSelector":{"autoInject":true,"enabled":false},"rewriteAppHTTPProbe":true},"telemetry":{"enabled":true,"v2":{"enabled":true,"metadataExchange":{"wasmEnabled":false},"prometheus":{"enabled":true,"wasmEnabled":false},"stackdriver":{"configOverride":{},"enabled":false,"logging":false,"monitoring":false,"topology":false}}}}}}
---
>       {"apiVersion":"install.istio.io/v1alpha1","kind":"IstioOperator","metadata":{"annotations":{"install.istio.io/ignoreReconcile":"true"},"creationTimestamp":"2020-12-21T11:52:59Z","generation":2,"managedFields":[{"apiVersion":"install.istio.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:install.istio.io/ignoreReconcile":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:addonComponents":{".":{},"f:istiocoredns":{".":{},"f:enabled":{}}},"f:components":{".":{},"f:base":{".":{},"f:enabled":{}},"f:cni":{".":{},"f:enabled":{}},"f:egressGateways":{},"f:ingressGateways":{},"f:istiodRemote":{".":{},"f:enabled":{}},"f:pilot":{".":{},"f:enabled":{},"f:k8s":{".":{},"f:env":{},"f:readinessProbe":{".":{},"f:httpGet":{".":{},"f:path":{},"f:port":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:strategy":{".":{},"f:rollingUpdate":{".":{},"f:maxSurge":{},"f:maxUnavailable":{}}}}}},"f:hub":{},"f:meshConfig":{".":{},"f:accessLogFile":{},"f:defaultConfig":{".":{},"f:proxyMetadata":{}},"f:enablePrometheusMerge":{}},"f:profile":{},"f:tag":{},"f:values":{".":{},"f:base":{".":{},"f:enableCRDTemplates":{},"f:validationURL":{}},"f:clusterResources":{},"f:gateways":{".":{},"f:istio-egressgateway":{".":{},"f:autoscaleEnabled":{},"f:env":{},"f:name":{},"f:secretVolumes":{},"f:type":{},"f:zvpn":{}},"f:istio-ingressgateway":{".":{},"f:autoscaleEnabled":{},"f:env":{},"f:name":{},"f:secretVolumes":{},"f:type":{},"f:zvpn":{}}},"f:global":{".":{},"f:arch":{".":{},"f:amd64":{},"f:ppc64le":{},"f:s390x":{}},"f:configValidation":{},"f:defaultNodeSelector":{},"f:defaultPodDisruptionBudget":{".":{},"f:enabled":{}},"f:defaultResources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:imagePullPolicy":{},"f:imagePullSecrets":{},"f:istioNamespace":{},"f:istiod":{".":{},"f:enableAnalysis":{}},"f:jwtPolicy":{},"f:logAsJson":{},"f:logging":{".":{},"f:level":{}},"f:meshExpansion":{".":{},"f:enabled":{},"f:useILB":{}},"f:meshNetworks":{},"f:mountMtlsCerts":{},"f:multiCluster":{".":{},"f:clusterName":{},"f:enabled":{}},"f:network":{},"f:omitSidecarInjectorConfigMap":{},"f:oneNamespace":{},"f:operatorManageWebhooks":{},"f:pilotCertProvider":{},"f:priorityClassName":{},"f:proxy":{".":{},"f:autoInject":{},"f:clusterDomain":{},"f:componentLogLevel":{},"f:enableCoreDump":{},"f:excludeIPRanges":{},"f:excludeInboundPorts":{},"f:excludeOutboundPorts":{},"f:image":{},"f:includeIPRanges":{},"f:logLevel":{},"f:privileged":{},"f:readinessFailureThreshold":{},"f:readinessInitialDelaySeconds":{},"f:readinessPeriodSeconds":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:statusPort":{},"f:tracer":{}},"f:proxy_init":{".":{},"f:image":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}}},"f:sds":{".":{},"f:token":{".":{},"f:aud":{}}},"f:sts":{".":{},"f:servicePort":{}},"f:tracer":{".":{},"f:datadog":{},"f:lightstep":{},"f:stackdriver":{},"f:zipkin":{}},"f:useMCP":{}},"f:istiocoredns":{".":{},"f:coreDNSImage":{},"f:coreDNSPluginImage":{},"f:coreDNSTag":{}},"f:istiodRemote":{".":{},"f:injectionURL":{}},"f:pilot":{".":{},"f:autoscaleEnabled":{},"f:autoscaleMax":{},"f:autoscaleMin":{},"f:configMap":{},"f:cpu":{".":{},"f:targetAverageUtilization":{}},"f:enableProtocolSniffingForInbound":{},"f:enableProtocolSniffingForOutbound":{},"f:env":{".":{},"f:ENABLE_LEGACY_FSGROUP_INJECTION":{}},"f:image":{},"f:keepaliveMaxServerConnectionAge":{},"f:nodeSelector":{},"f:replicaCount":{},"f:traceSampling":{}},"f:sidecarInjectorWebhook":{".":{},"f:enableNamespacesByDefault":{},"f:objectSelector":{".":{},"f:autoInject":{},"f:enabled":{}},"f:rewriteAppHTTPProbe":{}},"f:telemetry":{".":{},"f:enabled":{},"f:v2":{".":{},"f:enabled":{},"f:metadataExchange":{".":{},"f:wasmEnabled":{}},"f:prometheus":{".":{},"f:enabled":{},"f:wasmEnabled":{}},"f:stackdriver":{".":{},"f:configOverride":{},"f:enabled":{},"f:logging":{},"f:monitoring":{},"f:topology":{}}}}}}},"manager":"istioctl","operation":"Update","time":"2021-01-25T17:46:57Z"}],"name":"installed-state","namespace":"istio-system","resourceVersion":"335856","uid":"09a79cfd-9417-4de4-8b1c-4f8ba39189d0"},"spec":{"addonComponents":{"istiocoredns":{"enabled":false}},"components":{"base":{"enabled":true},"cni":{"enabled":false},"egressGateways":[{"enabled":true,"k8s":{"env":[{"name":"ISTIO_META_ROUTER_MODE","value":"standard"}],"hpaSpec":{"maxReplicas":5,"metrics":[{"resource":{"name":"cpu","targetAverageUtilization":80},"type":"Resource"}],"minReplicas":1,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"istio-egressgateway"}},"resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"40Mi"}},"service":{"ports":[{"name":"http2","port":80,"protocol":"TCP","targetPort":8080},{"name":"https","port":443,"protocol":"TCP","targetPort":8443},{"name":"tls","port":15443,"protocol":"TCP","targetPort":15443}]},"strategy":{"rollingUpdate":{"maxSurge":"100%","maxUnavailable":"25%"}}},"name":"istio-egressgateway"}],"ingressGateways":[{"enabled":true,"k8s":{"env":[{"name":"ISTIO_META_ROUTER_MODE","value":"standard"}],"hpaSpec":{"maxReplicas":5,"metrics":[{"resource":{"name":"cpu","targetAverageUtilization":80},"type":"Resource"}],"minReplicas":1,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"istio-ingressgateway"}},"resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"40Mi"}},"service":{"ports":[{"name":"status-port","port":15021,"protocol":"TCP","targetPort":15021},{"name":"http2","port":80,"protocol":"TCP","targetPort":8080},{"name":"https","port":443,"protocol":"TCP","targetPort":8443},{"name":"tcp","port":31400,"protocol":"TCP","targetPort":31400},{"name":"tls","port":15443,"protocol":"TCP","targetPort":15443}]},"strategy":{"rollingUpdate":{"maxSurge":"100%","maxUnavailable":"25%"}}},"name":"istio-ingressgateway"}],"istiodRemote":{"enabled":false},"pilot":{"enabled":true,"k8s":{"env":[{"name":"PILOT_TRACE_SAMPLING","value":"100"}],"readinessProbe":{"httpGet":{"path":"/ready","port":8080},"initialDelaySeconds":1,"periodSeconds":3,"timeoutSeconds":5},"resources":{"requests":{"cpu":"10m","memory":"100Mi"}},"strategy":{"rollingUpdate":{"maxSurge":"100%","maxUnavailable":"25%"}}}}},"hub":"docker.io/istio","meshConfig":{"accessLogFile":"/dev/stdout","defaultConfig":{"proxyMetadata":{}},"enablePrometheusMerge":true},"profile":"demo","tag":"1.8.2","values":{"base":{"enableCRDTemplates":false,"validationURL":""},"clusterResources":true,"gateways":{"istio-egressgateway":{"autoscaleEnabled":false,"env":{},"name":"istio-egressgateway","secretVolumes":[{"mountPath":"/etc/istio/egressgateway-certs","name":"egressgateway-certs","secretName":"istio-egressgateway-certs"},{"mountPath":"/etc/istio/egressgateway-ca-certs","name":"egressgateway-ca-certs","secretName":"istio-egressgateway-ca-certs"}],"type":"ClusterIP","zvpn":{}},"istio-ingressgateway":{"autoscaleEnabled":false,"env":{},"name":"istio-ingressgateway","secretVolumes":[{"mountPath":"/etc/istio/ingressgateway-certs","name":"ingressgateway-certs","secretName":"istio-ingressgateway-certs"},{"mountPath":"/etc/istio/ingressgateway-ca-certs","name":"ingressgateway-ca-certs","secretName":"istio-ingressgateway-ca-certs"}],"type":"LoadBalancer","zvpn":{}}},"global":{"arch":{"amd64":2,"ppc64le":2,"s390x":2},"configValidation":true,"defaultNodeSelector":{},"defaultPodDisruptionBudget":{"enabled":true},"defaultResources":{"requests":{"cpu":"10m"}},"imagePullPolicy":"","imagePullSecrets":[],"istioNamespace":"istio-system","istiod":{"enableAnalysis":false},"jwtPolicy":"third-party-jwt","logAsJson":false,"logging":{"level":"default:info"},"meshExpansion":{"enabled":false,"useILB":false},"meshNetworks":{},"mountMtlsCerts":false,"multiCluster":{"clusterName":"","enabled":false},"network":"","omitSidecarInjectorConfigMap":false,"oneNamespace":false,"operatorManageWebhooks":false,"pilotCertProvider":"istiod","priorityClassName":"","proxy":{"autoInject":"enabled","clusterDomain":"cluster.local","componentLogLevel":"misc:error","enableCoreDump":false,"excludeIPRanges":"","excludeInboundPorts":"","excludeOutboundPorts":"","image":"proxyv2","includeIPRanges":"*","logLevel":"warning","privileged":false,"readinessFailureThreshold":30,"readinessInitialDelaySeconds":1,"readinessPeriodSeconds":2,"resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"40Mi"}},"statusPort":15020,"tracer":"zipkin"},"proxy_init":{"image":"proxyv2","resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"10Mi"}}},"sds":{"token":{"aud":"istio-ca"}},"sts":{"servicePort":0},"tracer":{"datadog":{},"lightstep":{},"stackdriver":{},"zipkin":{}},"useMCP":false},"istiocoredns":{"coreDNSImage":"coredns/coredns","coreDNSPluginImage":"istio/coredns-plugin:0.2-istio-1.1","coreDNSTag":"1.6.2"},"istiodRemote":{"injectionURL":""},"pilot":{"autoscaleEnabled":false,"autoscaleMax":5,"autoscaleMin":1,"configMap":true,"cpu":{"targetAverageUtilization":80},"enableProtocolSniffingForInbound":true,"enableProtocolSniffingForOutbound":true,"env":{"ENABLE_LEGACY_FSGROUP_INJECTION":false},"image":"pilot","keepaliveMaxServerConnectionAge":"30m","nodeSelector":{},"replicaCount":1,"traceSampling":1},"sidecarInjectorWebhook":{"enableNamespacesByDefault":false,"objectSelector":{"autoInject":true,"enabled":false},"rewriteAppHTTPProbe":true},"telemetry":{"enabled":true,"v2":{"enabled":true,"metadataExchange":{"wasmEnabled":false},"prometheus":{"enabled":true,"wasmEnabled":false},"stackdriver":{"configOverride":{},"enabled":false,"logging":false,"monitoring":false,"topology":false}}}}}}
18d17
<           f:kubectl.kubernetes.io/last-applied-configuration: {}
258a258,266
>   - apiVersion: install.istio.io/v1alpha1
>     fieldsType: FieldsV1
>     fieldsV1:
>       f:metadata:
>         f:annotations:
>           f:kubectl.kubernetes.io/last-applied-configuration: {}
>     manager: kubectl-client-side-apply
>     operation: Update
>     time: "2021-01-26T10:30:01Z"
261c269
<   resourceVersion: "335856"
---
>   resourceVersion: "353467"
400,431d407
<     pilot:
<       jwksResolverExtraRootCA: |
<         -----BEGIN CERTIFICATE-----
<         MIIFFDCCAvygAwIBAgIUR5/JByU16/FqSqqhiZ1Lk5m0cDYwDQYJKoZIhvcNAQEL
<         BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjAxMjIx
<         MTEzNjI4WhcNMzAxMjE5MTEzNjI4WjAiMQ4wDAYDVQQKDAVJc3RpbzEQMA4GA1UE
<         AwwHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAJ7peieP
<         6beOn8815tWDwW7Ro7SMZzlOJxAYZFxwtqi9HsKYAOtMcbjYMf0bdWES6WxmXI3N
<         NSbVtgZeQESG89ZL5l+CoCSPht5yBI7QnvbSO99qxxF8DawGlmSXBKTaReIro3Z8
<         mBKFT9fNT+JK+ORZR5RI3juaqKElL3uRnSXTOT51SGjJqzVIJ+iD7cuBYTta1vXl
<         eDwM2CtWW9hKYMbvEwhP6sEr1JlQsuPJKrqerQt/8uiH3oF5iZ8GND12iMlG8gzg
<         et9DweUX5cR9AwrBON7+SfEoAnWObAcY19h+lDCDujWyHJMI5vPe09Ampoa5T7+q
<         WtMM2Fjm4XvFEu4kwthqT0mxmtXvtlPlBTO/YLZ+qhVjAOS5K7HU47Bu51Q0hOky
<         irxkU7kXZrvSekXWYFimytmjjN2nCAhbeXG8bmJmOxBcgDHMea1SFPyHQHIsQNRW
<         91Hg2VNk9ooqYyMLxuYy4TdAYUg+t+TNLGsAYkY4bWrII9D/CkvBlMiFYerbS6ds
<         4ee7n6jJNhmzJcg/ELoocvPBaDLjll6mBjCVboSycwRHN5Fd96Jc+AWhHUTxCv1z
<         252Jql8cKR+YGok4xNC6qWR7qIPKCOpac7gRfcnD15eRW5ynhqy82tHQY7m9Ss5g
<         4vcEHjnFLYDN7LN6Od9MmxTsxlFq7DdNzG7rAgMBAAGjQjBAMB0GA1UdDgQWBBSM
<         wXc6WuZO7HtmIGaRpGd4l/uitTAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE
<         AwIC5DANBgkqhkiG9w0BAQsFAAOCAgEAdSHQ6EG7VUMMcysDgcvxWPKWgbliFtWq
<         qJ4xMg8CI3Ya96zcUHdvPayzkIde93KEyt6GUzUYOjuo4cmLswDr0Jxl2CMcEchY
<         cLW9oW+4FkyhE4C7/6rGbDmYNUu2GyJJ6Ut9px+jGdAO9uOxmk1jsGD+fahkTi7H
<         ZXpQC48iEublkUZHFK2jzsWk4920lWlYLCOzfqRQH+btXQ5Wlmy/yXfpqwqk9WQd
<         AYw1frXhULaUp+LYiICfQjvYiK9/zNCrLM865MEZmGVmzMGjRhSJxNA9ZlGI/yEy
<         1Arx51Ci3jSFMjJBpVGM5TGv0PUvGmbMJBp/oRiiQr/BxOkeITrvdGT/QyYOdWGT
<         dEgBW9pW6wAurq7qUaaR3IKxdev2FsvF+1rWwENk1SPnyA2+wGGPLhAnDn6ED/uk
<         1hBZWQiPIkCxwHmkRTXjAJceBtV/e44HSdubEsBrcVr+cd0BPFqqDwMz19vY4FYo
<         /fcyUAwXuh4MUDQO680PxFW5sOvQ6c8rd7ybybj7aiwk8EO/fesZPfOrwYJtNsaY
<         u5Qw/LxTmN9MAkr280FFMWFeN+WfcsElI7enTURLDfQbVBtKbdT5HwfoaixJrQqW
<         6Hg4pCH9PgCy2RAaSNVUriRqwc8xSLvs/P8pWtGu/1zaocyvo9I9u0oJPDslrlWY
<         SKCsExuPHWs=
<         -----END CERTIFICATE-----
sysop@hdev:~/software/hproject$

What I'm doing wrong?

Sorry, I've seen that the issue is assigned to @xulingqing

Please see the previous two posts.

Hi @xulingqing One step haed but still certificate not accepted. I was able to insert the certificate after the fisrt "pilot:" occurrence (line 379) Now the istio operator is:

sysop@hdev:~/software/hproject$ kubectl describe istiooperator -n istio-system
Name:         installed-state
Namespace:    istio-system
Labels:       <none>
Annotations:  install.istio.io/ignoreReconcile: true
API Version:  install.istio.io/v1alpha1
Kind:         IstioOperator
Metadata:
  Creation Timestamp:  2020-12-21T11:52:59Z
  Generation:          3
  Managed Fields:
    API Version:  install.istio.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:install.istio.io/ignoreReconcile:
      f:spec:
        .:
        f:addonComponents:
          .:
          f:istiocoredns:
            .:
            f:enabled:
        f:components:
          .:
          f:base:
            .:
            f:enabled:
          f:cni:
            .:
            f:enabled:
          f:egressGateways:
          f:ingressGateways:
          f:istiodRemote:
            .:
            f:enabled:
          f:pilot:
            .:
            f:enabled:
            f:k8s:
              .:
              f:env:
              f:readinessProbe:
                .:
                f:httpGet:
                  .:
                  f:path:
                  f:port:
                f:initialDelaySeconds:
                f:periodSeconds:
                f:timeoutSeconds:
              f:resources:
                .:
                f:requests:
                  .:
                  f:cpu:
                  f:memory:
              f:strategy:
                .:
                f:rollingUpdate:
                  .:
                  f:maxSurge:
                  f:maxUnavailable:
        f:hub:
        f:meshConfig:
          .:
          f:accessLogFile:
          f:defaultConfig:
            .:
            f:proxyMetadata:
          f:enablePrometheusMerge:
        f:profile:
        f:tag:
        f:values:
          .:
          f:base:
            .:
            f:enableCRDTemplates:
            f:validationURL:
          f:clusterResources:
          f:gateways:
            .:
            f:istio-egressgateway:
              .:
              f:autoscaleEnabled:
              f:env:
              f:name:
              f:secretVolumes:
              f:type:
              f:zvpn:
            f:istio-ingressgateway:
              .:
              f:autoscaleEnabled:
              f:env:
              f:name:
              f:secretVolumes:
              f:type:
              f:zvpn:
          f:global:
            .:
            f:arch:
              .:
              f:amd64:
              f:ppc64le:
              f:s390x:
            f:configValidation:
            f:defaultNodeSelector:
            f:defaultPodDisruptionBudget:
              .:
              f:enabled:
            f:defaultResources:
              .:
              f:requests:
                .:
                f:cpu:
            f:imagePullPolicy:
            f:imagePullSecrets:
            f:istioNamespace:
            f:istiod:
              .:
              f:enableAnalysis:
            f:jwtPolicy:
            f:logAsJson:
            f:logging:
              .:
              f:level:
            f:meshExpansion:
              .:
              f:enabled:
              f:useILB:
            f:meshNetworks:
            f:mountMtlsCerts:
            f:multiCluster:
              .:
              f:clusterName:
              f:enabled:
            f:network:
            f:omitSidecarInjectorConfigMap:
            f:oneNamespace:
            f:operatorManageWebhooks:
            f:pilotCertProvider:
            f:priorityClassName:
            f:proxy:
              .:
              f:autoInject:
              f:clusterDomain:
              f:componentLogLevel:
              f:enableCoreDump:
              f:excludeIPRanges:
              f:excludeInboundPorts:
              f:excludeOutboundPorts:
              f:image:
              f:includeIPRanges:
              f:logLevel:
              f:privileged:
              f:readinessFailureThreshold:
              f:readinessInitialDelaySeconds:
              f:readinessPeriodSeconds:
              f:resources:
                .:
                f:limits:
                  .:
                  f:cpu:
                  f:memory:
                f:requests:
                  .:
                  f:cpu:
                  f:memory:
              f:statusPort:
              f:tracer:
            f:proxy_init:
              .:
              f:image:
              f:resources:
                .:
                f:limits:
                  .:
                  f:cpu:
                  f:memory:
                f:requests:
                  .:
                  f:cpu:
                  f:memory:
            f:sds:
              .:
              f:token:
                .:
                f:aud:
            f:sts:
              .:
              f:servicePort:
            f:tracer:
              .:
              f:datadog:
              f:lightstep:
              f:stackdriver:
              f:zipkin:
            f:useMCP:
          f:istiocoredns:
            .:
            f:coreDNSImage:
            f:coreDNSPluginImage:
            f:coreDNSTag:
          f:istiodRemote:
            .:
            f:injectionURL:
          f:pilot:
            .:
            f:autoscaleEnabled:
            f:autoscaleMax:
            f:autoscaleMin:
            f:configMap:
            f:cpu:
              .:
              f:targetAverageUtilization:
            f:enableProtocolSniffingForInbound:
            f:enableProtocolSniffingForOutbound:
            f:env:
              .:
              f:ENABLE_LEGACY_FSGROUP_INJECTION:
            f:image:
            f:keepaliveMaxServerConnectionAge:
            f:nodeSelector:
            f:replicaCount:
            f:traceSampling:
          f:sidecarInjectorWebhook:
            .:
            f:enableNamespacesByDefault:
            f:objectSelector:
              .:
              f:autoInject:
              f:enabled:
            f:rewriteAppHTTPProbe:
          f:telemetry:
            .:
            f:enabled:
            f:v2:
              .:
              f:enabled:
              f:metadataExchange:
                .:
                f:wasmEnabled:
              f:prometheus:
                .:
                f:enabled:
                f:wasmEnabled:
              f:stackdriver:
                .:
                f:configOverride:
                f:enabled:
                f:logging:
                f:monitoring:
                f:topology:
    Manager:      istioctl
    Operation:    Update
    Time:         2021-01-25T17:46:57Z
    API Version:  install.istio.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:components:
          f:pilot:
            f:jwksResolverExtraRootCA:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-01-26T19:24:44Z
  Resource Version:  394956
  UID:               09a79cfd-9417-4de4-8b1c-4f8ba39189d0
Spec:
  Addon Components:
    Istiocoredns:
      Enabled:  false
  Components:
    Base:
      Enabled:  true
    Cni:
      Enabled:  false
    Egress Gateways:
      Enabled:  true
      k8s:
        Env:
          Name:   ISTIO_META_ROUTER_MODE
          Value:  standard
        Hpa Spec:
          Max Replicas:  5
          Metrics:
            Resource:
              Name:                        cpu
              Target Average Utilization:  80
            Type:                          Resource
          Min Replicas:                    1
          Scale Target Ref:
            API Version:  apps/v1
            Kind:         Deployment
            Name:         istio-egressgateway
        Resources:
          Limits:
            Cpu:     2000m
            Memory:  1024Mi
          Requests:
            Cpu:     10m
            Memory:  40Mi
        Service:
          Ports:
            Name:         http2
            Port:         80
            Protocol:     TCP
            Target Port:  8080
            Name:         https
            Port:         443
            Protocol:     TCP
            Target Port:  8443
            Name:         tls
            Port:         15443
            Protocol:     TCP
            Target Port:  15443
        Strategy:
          Rolling Update:
            Max Surge:        100%
            Max Unavailable:  25%
      Name:                   istio-egressgateway
    Ingress Gateways:
      Enabled:  true
      k8s:
        Env:
          Name:   ISTIO_META_ROUTER_MODE
          Value:  standard
        Hpa Spec:
          Max Replicas:  5
          Metrics:
            Resource:
              Name:                        cpu
              Target Average Utilization:  80
            Type:                          Resource
          Min Replicas:                    1
          Scale Target Ref:
            API Version:  apps/v1
            Kind:         Deployment
            Name:         istio-ingressgateway
        Resources:
          Limits:
            Cpu:     2000m
            Memory:  1024Mi
          Requests:
            Cpu:     10m
            Memory:  40Mi
        Service:
          Ports:
            Name:         status-port
            Port:         15021
            Protocol:     TCP
            Target Port:  15021
            Name:         http2
            Port:         80
            Protocol:     TCP
            Target Port:  8080
            Name:         https
            Port:         443
            Protocol:     TCP
            Target Port:  8443
            Name:         tcp
            Port:         31400
            Protocol:     TCP
            Target Port:  31400
            Name:         tls
            Port:         15443
            Protocol:     TCP
            Target Port:  15443
        Strategy:
          Rolling Update:
            Max Surge:        100%
            Max Unavailable:  25%
      Name:                   istio-ingressgateway
    Istiod Remote:
      Enabled:  false
    Pilot:
      Enabled:                      true
      Jwks Resolver Extra Root CA:  -----BEGIN CERTIFICATE-----
MIIFFDCCAvygAwIBAgIUR5/JByU16/FqSqqhiZ1Lk5m0cDYwDQYJKoZIhvcNAQEL
BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjAxMjIx
MTEzNjI4WhcNMzAxMjE5MTEzNjI4WjAiMQ4wDAYDVQQKDAVJc3RpbzEQMA4GA1UE
AwwHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAJ7peieP
6beOn8815tWDwW7Ro7SMZzlOJxAYZFxwtqi9HsKYAOtMcbjYMf0bdWES6WxmXI3N
NSbVtgZeQESG89ZL5l+CoCSPht5yBI7QnvbSO99qxxF8DawGlmSXBKTaReIro3Z8
mBKFT9fNT+JK+ORZR5RI3juaqKElL3uRnSXTOT51SGjJqzVIJ+iD7cuBYTta1vXl
eDwM2CtWW9hKYMbvEwhP6sEr1JlQsuPJKrqerQt/8uiH3oF5iZ8GND12iMlG8gzg
et9DweUX5cR9AwrBON7+SfEoAnWObAcY19h+lDCDujWyHJMI5vPe09Ampoa5T7+q
WtMM2Fjm4XvFEu4kwthqT0mxmtXvtlPlBTO/YLZ+qhVjAOS5K7HU47Bu51Q0hOky
irxkU7kXZrvSekXWYFimytmjjN2nCAhbeXG8bmJmOxBcgDHMea1SFPyHQHIsQNRW
91Hg2VNk9ooqYyMLxuYy4TdAYUg+t+TNLGsAYkY4bWrII9D/CkvBlMiFYerbS6ds
4ee7n6jJNhmzJcg/ELoocvPBaDLjll6mBjCVboSycwRHN5Fd96Jc+AWhHUTxCv1z
252Jql8cKR+YGok4xNC6qWR7qIPKCOpac7gRfcnD15eRW5ynhqy82tHQY7m9Ss5g
4vcEHjnFLYDN7LN6Od9MmxTsxlFq7DdNzG7rAgMBAAGjQjBAMB0GA1UdDgQWBBSM
wXc6WuZO7HtmIGaRpGd4l/uitTAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE
AwIC5DANBgkqhkiG9w0BAQsFAAOCAgEAdSHQ6EG7VUMMcysDgcvxWPKWgbliFtWq
qJ4xMg8CI3Ya96zcUHdvPayzkIde93KEyt6GUzUYOjuo4cmLswDr0Jxl2CMcEchY
cLW9oW+4FkyhE4C7/6rGbDmYNUu2GyJJ6Ut9px+jGdAO9uOxmk1jsGD+fahkTi7H
ZXpQC48iEublkUZHFK2jzsWk4920lWlYLCOzfqRQH+btXQ5Wlmy/yXfpqwqk9WQd
AYw1frXhULaUp+LYiICfQjvYiK9/zNCrLM865MEZmGVmzMGjRhSJxNA9ZlGI/yEy
1Arx51Ci3jSFMjJBpVGM5TGv0PUvGmbMJBp/oRiiQr/BxOkeITrvdGT/QyYOdWGT
dEgBW9pW6wAurq7qUaaR3IKxdev2FsvF+1rWwENk1SPnyA2+wGGPLhAnDn6ED/uk
1hBZWQiPIkCxwHmkRTXjAJceBtV/e44HSdubEsBrcVr+cd0BPFqqDwMz19vY4FYo
/fcyUAwXuh4MUDQO680PxFW5sOvQ6c8rd7ybybj7aiwk8EO/fesZPfOrwYJtNsaY
u5Qw/LxTmN9MAkr280FFMWFeN+WfcsElI7enTURLDfQbVBtKbdT5HwfoaixJrQqW
6Hg4pCH9PgCy2RAaSNVUriRqwc8xSLvs/P8pWtGu/1zaocyvo9I9u0oJPDslrlWY
SKCsExuPHWs=
-----END CERTIFICATE-----

      k8s:
        Env:
          Name:   PILOT_TRACE_SAMPLING
          Value:  100
        Readiness Probe:
          Http Get:
            Path:                 /ready
            Port:                 8080
          Initial Delay Seconds:  1
          Period Seconds:         3
          Timeout Seconds:        5
        Resources:
          Requests:
            Cpu:     10m
            Memory:  100Mi
        Strategy:
          Rolling Update:
            Max Surge:        100%
            Max Unavailable:  25%
  Hub:                        docker.io/istio
  Mesh Config:
    Access Log File:  /dev/stdout
    Default Config:
      Proxy Metadata:
    Enable Prometheus Merge:  true
  Profile:                    demo
  Tag:                        1.8.2
  Values:
    Base:
      Enable CRD Templates:  false
      Validation URL:        
    Cluster Resources:       true
    Gateways:
      Istio - Egressgateway:
        Autoscale Enabled:  false
        Env:
        Name:  istio-egressgateway
        Secret Volumes:
          Mount Path:   /etc/istio/egressgateway-certs
          Name:         egressgateway-certs
          Secret Name:  istio-egressgateway-certs
          Mount Path:   /etc/istio/egressgateway-ca-certs
          Name:         egressgateway-ca-certs
          Secret Name:  istio-egressgateway-ca-certs
        Type:           ClusterIP
        Zvpn:
      Istio - Ingressgateway:
        Autoscale Enabled:  false
        Env:
        Name:  istio-ingressgateway
        Secret Volumes:
          Mount Path:   /etc/istio/ingressgateway-certs
          Name:         ingressgateway-certs
          Secret Name:  istio-ingressgateway-certs
          Mount Path:   /etc/istio/ingressgateway-ca-certs
          Name:         ingressgateway-ca-certs
          Secret Name:  istio-ingressgateway-ca-certs
        Type:           LoadBalancer
        Zvpn:
    Global:
      Arch:
        amd64:            2
        ppc64le:          2
        s390x:            2
      Config Validation:  true
      Default Node Selector:
      Default Pod Disruption Budget:
        Enabled:  true
      Default Resources:
        Requests:
          Cpu:            10m
      Image Pull Policy:  
      Image Pull Secrets:
      Istio Namespace:  istio-system
      Istiod:
        Enable Analysis:  false
      Jwt Policy:         third-party-jwt
      Log As Json:        false
      Logging:
        Level:  default:info
      Mesh Expansion:
        Enabled:  false
        Use ILB:  false
      Mesh Networks:
      Mount Mtls Certs:  false
      Multi Cluster:
        Cluster Name:                    
        Enabled:                         false
      Network:                           
      Omit Sidecar Injector Config Map:  false
      One Namespace:                     false
      Operator Manage Webhooks:          false
      Pilot Cert Provider:               istiod
      Priority Class Name:               
      Proxy:
        Auto Inject:                      enabled
        Cluster Domain:                   cluster.local
        Component Log Level:              misc:error
        Enable Core Dump:                 false
        Exclude IP Ranges:                
        Exclude Inbound Ports:            
        Exclude Outbound Ports:           
        Image:                            proxyv2
        Include IP Ranges:                *
        Log Level:                        warning
        Privileged:                       false
        Readiness Failure Threshold:      30
        Readiness Initial Delay Seconds:  1
        Readiness Period Seconds:         2
        Resources:
          Limits:
            Cpu:     2000m
            Memory:  1024Mi
          Requests:
            Cpu:      10m
            Memory:   40Mi
        Status Port:  15020
        Tracer:       zipkin
      proxy_init:
        Image:  proxyv2
        Resources:
          Limits:
            Cpu:     2000m
            Memory:  1024Mi
          Requests:
            Cpu:     10m
            Memory:  10Mi
      Sds:
        Token:
          Aud:  istio-ca
      Sts:
        Service Port:  0
      Tracer:
        Datadog:
        Lightstep:
        Stackdriver:
        Zipkin:
      Use MCP:  false
    Istiocoredns:
      Core DNS Image:         coredns/coredns
      Core DNS Plugin Image:  istio/coredns-plugin:0.2-istio-1.1
      Core DNS Tag:           1.6.2
    Istiod Remote:
      Injection URL:  
    Pilot:
      Autoscale Enabled:  false
      Autoscale Max:      5
      Autoscale Min:      1
      Config Map:         true
      Cpu:
        Target Average Utilization:           80
      Enable Protocol Sniffing For Inbound:   true
      Enable Protocol Sniffing For Outbound:  true
      Env:
        ENABLE_LEGACY_FSGROUP_INJECTION:    false
      Image:                                pilot
      Keepalive Max Server Connection Age:  30m
      Node Selector:
      Replica Count:   1
      Trace Sampling:  1
    Sidecar Injector Webhook:
      Enable Namespaces By Default:  false
      Object Selector:
        Auto Inject:           true
        Enabled:               false
      Rewrite App HTTP Probe:  true
    Telemetry:
      Enabled:  true
      v2:
        Enabled:  true
        Metadata Exchange:
          Wasm Enabled:  false
        Prometheus:
          Enabled:       true
          Wasm Enabled:  false
        Stackdriver:
          Config Override:
          Enabled:     false
          Logging:     false
          Monitoring:  false
          Topology:    false
Events:                <none>
sysop@hdev:~/software/hproject$

Still in the istiod log appear the " x509: certificate signed by unknown authority" error

2021-01-27T10:38:31.438413Z info    ads EDS: PUSH for node:istio-ingressgateway-68c86b9fc8-dq4qk.istio-system resources:37 empty:0 cached:37/37
2021-01-27T10:38:31.495824Z error   model   Failed to fetch public key from "https://k6k.h.net/auth/realms/hproject/protocol/openid-connect/certs": Get "https://k6k.h.net/auth/realms/hproject/protocol/openid-connect/certs": x509: certificate signed by unknown authority
2021-01-27T10:38:31.495869Z error   Failed to fetch jwt public key from "https://k6k.h.net/auth/realms/hproject/protocol/openid-connect/certs": Get "https://k6k.h.net/auth/realms/hproject/protocol/openid-connect/certs": x509: certificate signed by unknown authority
2021-01-27T10:38:31.511834Z error   model   Failed to fetch public key from "https://k6k.h.net/auth/realms/hproject/protocol/openid-connect/certs": Get "https://k6k.h.net/auth/realms/hproject/protocol/openid-connect/certs": x509: certificate signed by unknown authority
2021-01-27T10:38:31.511872Z error   Failed to fetch jwt public key from "https://k6k.h.net/auth/realms/hproject/protocol/openid-connect/certs": Get "https://k6k.h.net/auth/realms/hproject/protocol/openid-connect/certs": x509: certificate signed by unknown authority
2021-01-27T10:38:31.512387Z info    ads LDS: PUSH for node:istio-ingressgateway-68c86b9fc8-dq4qk.istio-system resources:1

Mybe I'm inserting the certificate in the wrong place?

What can I do?

Hi @xulingqing do you prefer that I open a new issue with title "How to set jwksResolverExtraRootCA"?

Hi @xulingqing @yangminzhu any news about this problem?

Hi @xulingqing @yangminzhu I have found a solution: Install Istio 1.9.1 using istio operator The configuration file (istio-operator) contains

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: demo
  meshConfig:
    accessLogEncoding: TEXT
    accessLogFile: "/dev/stdout"
    accessLogFormat: ""
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  values:
    pilot:
      jwksResolverExtraRootCA: |
        -----BEGIN CERTIFICATE-----
        MIIEKTCCAxGgAwIBAgIUNjT4liTyGcGTewYUb2NZj4oWMGowDQYJKoZIhvcNAQEL
        BQAwgaMxCzAJBgNVBAYTAklUMQ4wDAYDVQQIDAVJdGFseTENMAsGA1UEBwwEUm9t
        ZTEcMBoGA1UECgwTQnVzaWNvIE1pcnRvIFNpbHZpbzETMBEGA1UECwwKTGFib3Jh
        dG9yeTEcMBoGA1UEAwwTQnVzaWNvIE1pcnRvIFNpbHZpbzEkMCIGCSqGSIb3DQEJ
        ARYVbWlydG9idXNpY29AZ21haWwuY29tMB4XDTIxMDIxOTExNDA1N1oXDTMxMDIx
        NzExNDA1N1owgaMxCzAJBgNVBAYTAklUMQ4wDAYDVQQIDAVJdGFseTENMAsGA1UE
        BwwEUm9tZTEcMBoGA1UECgwTQnVzaWNvIE1pcnRvIFNpbHZpbzETMBEGA1UECwwK
        TGFib3JhdG9yeTEcMBoGA1UEAwwTQnVzaWNvIE1pcnRvIFNpbHZpbzEkMCIGCSqG
        SIb3DQEJARYVbWlydG9idXNpY29AZ21haWwuY29tMIIBIjANBgkqhkiG9w0BAQEF
        AAOCAQ8AMIIBCgKCAQEAtt+v2C9p60rx1Q/yOQsgsis/dBNAo4efFlyN0Ibs9ts4
        a0LRQp4EwZpv9+tysVqGZvN4fJ99mdyiJHiFlchMfq4t+OzOHnym7Yi5khHS5/rv
        TwvwD+1igMny1FybVOxSlfdZGF5mhgRFD6mQod/hix5QJgegmygEhj0VV/i2rZhH
        FW0oMR1smLfALQQZhGJ//TgCjNlpK2D6zlxEXIr+QLyAje+kQyyqkJefY/Vggg9m
        gRFV7gu3MKKGE5B+ESfqhZbUqsKz/rsxs2L23Selp3FM+DhKIC8DM06dgAh7DYUQ
        IeIe9HT7+RTTs3KM7ArDXrF+BF+D8O/a4D3YBDhHswIDAQABo1MwUTAdBgNVHQ4E
        FgQUGHqUwQ6vRxvpB7+MCUEAwmUgbtAwHwYDVR0jBBgwFoAUGHqUwQ6vRxvpB7+M
        CUEAwmUgbtAwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAVThR
        2NmNuey07twCW4/B8v6zOCeO/n8Z+waRw1XK9XmA+QPUTi+bLKvfx+7RVgaZD6SR
        EQHMCshGD7In5PbSBsrp6ocmCvcopd2iqvt2GLvJHuZy7hI+RgaMgQo9hhThHf9e
        FoW3C41Mm3ofvUubIKLEFKnCvxOsD+Ayyg69pGwNM+PQK1XvXWWYm8eroPICxriq
        8ULUgAc+leGiHbAKSXGLj6U/njyRzkxAmXzkierT1DFDHa9sZst7nCaSycKY7rBj
        GU2xRTOpYrQHcsaBZBjTT8/ag2IasCzFVeZ5+bMmTaDus5QT3tIubR8ukTx5jf0S
        BiQTL+/xni6Fxkrl3Q==
        -----END CERTIFICATE-----

And I installed with:

kubectl create ns istio-system
kubectl apply -f istio-operator

Now I can use the authentication file

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name:  m-ra
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "https://k6k.m.net/auth/realms/m-project"
    jwksUri: "https://k6k.m.net/auth/realms/m-project/protocol/openid-connect/certs"
    forwardOriginalToken: true
    outputPayloadToHeader: x-jwt-payload

For me the issue can be closed.

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-01-14. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

This is still not working with istioctl 1.19.1. I am unbale to push jwksResolverExtraRootCA

meshConfig: outboundTrafficPolicy: mode: REGISTRY_ONLY values: pilot: jwksResolverExtraRootCA: | -----BEGIN CERTIFICATE----- -----END CERTIFICATE-----

istio / istio

How to set a private CA certificate in istiod? #29366