unknown field "controlPlaneAuthPolicy" in istio_proxy_v1_config.ProxyConfig

[jpkrohling]

When trying to run kube-inject into a Kubernetes template, the following message is shown:

unknown field "controlPlaneAuthPolicy" in istio_proxy_v1_config.ProxyConfig

This can be reproduced by using Jaeger's template files for Kubernetes, located here: https://github.com/jaegertracing/jaeger-kubernetes

How to reproduce:

$ minikube start --vm-driver kvm --cpus 2 --memory 8192 --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota" --kubernetes-version=v1.8.0
$ kubectl create -f production/cassandra.yml
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/install/kubernetes/istio-auth.yaml
$ istioctl kube-inject -f jaeger-production-template.yml -i istio-system

The last command yields:

Error: could not read valid configmap "istio" from namespace  "istio-system": failed to convert to proto. unknown field "controlPlaneAuthPolicy" in istio_proxy_v1_config.ProxyConfig - Re-run kube-inject with `-i <istioSystemNamespace> and ensure valid MeshConfig exists

Version information:

$ istioctl version
Version: 0.2.10
GitRevision: f27f2803f59994367c1cca47467c362b1702d605
GitBranch: release-0.2
User: sebastienvas@ee792364cfc2
GolangVersion: go1.8.3

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6+a08f5eeb62", GitCommit:"c84beff", GitTreeState:"clean", BuildDate:"2017-10-20T14:41:29Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

And this is the configuration that is mentioned by the inject command:

kubectl get configmap -n istio-system istio -o yaml:

apiVersion: v1
data:
  mesh: |-
    # Uncomment the following line to enable mutual TLS between proxies
    authPolicy: MUTUAL_TLS
    #
    # Set the following variable to true to disable policy checks by the Mixer.
    # Note that metrics will still be reported to the Mixer.
    disablePolicyChecks: false
    # Set enableTracing to false to disable request tracing.
    enableTracing: true
    #
    # To disable the mixer completely (including metrics), comment out
    # the following line
    mixerAddress: istio-mixer.istio-system:15004
    # This is the ingress service name, update if you used a different name
    ingressService: istio-ingress
    #
    # Along with discoveryRefreshDelay, this setting determines how
    # frequently should Envoy fetch and update its internal configuration
    # from Istio Pilot. Lower refresh delay results in higher CPU
    # utilization and potential performance loss in exchange for faster
    # convergence. Tweak this value according to your setup.
    rdsRefreshDelay: 1s
    #
    defaultConfig:
      # NOTE: If you change any values in this section, make sure to make
      # the same changes in start up args in istio-ingress pods.
      # See rdsRefreshDelay for explanation about this setting.
      discoveryRefreshDelay: 1s
      #
      # TCP connection timeout between Envoy & the application, and between Envoys.
      connectTimeout: 10s
      #
      ### ADVANCED SETTINGS #############
      # Where should envoy's configuration be stored in the istio-proxy container
      configPath: "/etc/istio/proxy"
      binaryPath: "/usr/local/bin/envoy"
      # The pseudo service name used for Envoy.
      serviceCluster: istio-proxy
      # These settings that determine how long an old Envoy
      # process should be kept alive after an occasional reload.
      drainDuration: 45s
      parentShutdownDuration: 1m0s
      #
      # Port where Envoy listens (on local host) for admin commands
      # You can exec into the istio-proxy container in a pod and
      # curl the admin port (curl http://localhost:15000/) to obtain
      # diagnostic information from Envoy. See
      # https://lyft.github.io/envoy/docs/operations/admin.html
      # for more details
      proxyAdminPort: 15000
      #
      # Address where Istio Pilot service is running
      discoveryAddress: istio-pilot.istio-system:15003
      #
      # Zipkin trace collector
      zipkinAddress: zipkin.istio-system:9411
      #
      # Statsd metrics collector. Istio mixer exposes a UDP endpoint
      # to collect and convert statsd metrics into Prometheus metrics.
      statsdUdpAddress: istio-mixer.istio-system:9125
      # Uncomment the following line to enable mutual TLS authentication between
      # sidecars and istio control plane.
      controlPlaneAuthPolicy: MUTUAL_TLS
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"mesh":"# Uncomment the following line to enable mutual TLS between proxies\nauthPolicy: MUTUAL_TLS\n#\n# Set the following variable to true to disable policy checks by the Mixer.\n# Note that metrics will still be reported to the Mixer.\ndisablePolicyChecks: false\n# Set enableTracing to false to disable request tracing.\nenableTracing: true\n#\n# To disable the mixer completely (including metrics), comment out\n# the following line\nmixerAddress: istio-mixer.istio-system:15004\n# This is the ingress service name, update if you used a different name\ningressService: istio-ingress\n#\n# Along with discoveryRefreshDelay, this setting determines how\n# frequently should Envoy fetch and update its internal configuration\n# from Istio Pilot. Lower refresh delay results in higher CPU\n# utilization and potential performance loss in exchange for faster\n# convergence. Tweak this value according to your setup.\nrdsRefreshDelay: 1s\n#\ndefaultConfig:\n  # NOTE: If you change any values in this section, make sure to make\n  # the same changes in start up args in istio-ingress pods.\n  # See rdsRefreshDelay for explanation about this setting.\n  discoveryRefreshDelay: 1s\n  #\n  # TCP connection timeout between Envoy \u0026 the application, and between Envoys.\n  connectTimeout: 10s\n  #\n  ### ADVANCED SETTINGS #############\n  # Where should envoy's configuration be stored in the istio-proxy container\n  configPath: \"/etc/istio/proxy\"\n  binaryPath: \"/usr/local/bin/envoy\"\n  # The pseudo service name used for Envoy.\n  serviceCluster: istio-proxy\n  # These settings that determine how long an old Envoy\n  # process should be kept alive after an occasional reload.\n  drainDuration: 45s\n  parentShutdownDuration: 1m0s\n  #\n  # Port where Envoy listens (on local host) for admin commands\n  # You can exec into the istio-proxy container in a pod and\n  # curl the admin port (curl http://localhost:15000/) to obtain\n  # diagnostic information from Envoy. See\n  # https://lyft.github.io/envoy/docs/operations/admin.html\n  # for more details\n  proxyAdminPort: 15000\n  #\n  # Address where Istio Pilot service is running\n  discoveryAddress: istio-pilot.istio-system:15003\n  #\n  # Zipkin trace collector\n  zipkinAddress: zipkin.istio-system:9411\n  #\n  # Statsd metrics collector. Istio mixer exposes a UDP endpoint\n  # to collect and convert statsd metrics into Prometheus metrics.\n  statsdUdpAddress: istio-mixer.istio-system:9125\n  # Uncomment the following line to enable mutual TLS authentication between\n  # sidecars and istio control plane.\n  controlPlaneAuthPolicy: MUTUAL_TLS"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"istio","namespace":"istio-system"}}
  creationTimestamp: 2017-11-03T16:13:30Z
  name: istio
  namespace: istio-system
  resourceVersion: "743"
  selfLink: /api/v1/namespaces/istio-system/configmaps/istio
  uid: ee86b9e9-c0b1-11e7-9c1f-5254009962d2

[mandarjog]

That field was added as part of securing Istio components work about two weeks ago and it is available in master.

https://github.com/istio/api/commit/d80e4d55d86452dbb5de8ee24b37bf5c5124a9bc

Are you trying to use this file with a 0.2.x release? You can comment out that line from the config map and things should work.

• Mandar

On Fri, Nov 3, 2017 at 9:41 AM, Juraci Paixão Kröhling < notifications@github.com> wrote:

When trying to run kube-inject into a Kubernetes template, the following message is shown:

unknown field "controlPlaneAuthPolicy" in istio_proxy_v1_config.ProxyConfig

This can be reproduced by using Jaeger's template files for Kubernetes, located here: https://github.com/jaegertracing/jaeger-kubernetes

How to reproduce:

$ minikube start --vm-driver kvm --cpus 2 --memory 8192 --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota" --kubernetes-version=v1.8.0 $ kubectl create -f production/cassandra.yml $ kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/install/kubernetes/istio-auth.yaml $ istioctl kube-inject -f jaeger-production-template.yml -i istio-system

The last command yields:

Error: could not read valid configmap "istio" from namespace "istio-system": failed to convert to proto. unknown field "controlPlaneAuthPolicy" in istio_proxy_v1_config.ProxyConfig - Re-run kube-inject with `-i and ensure valid MeshConfig exists

Version information:

$ istioctl version Version: 0.2.10 GitRevision: f27f2803f59994367c1cca47467c362b1702d605 GitBranch: release-0.2 User: sebastienvas@ee792364cfc2 GolangVersion: go1.8.3

$ kubectl version Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6+a08f5eeb62", GitCommit:"c84beff", GitTreeState:"clean", BuildDate:"2017-10-20T14:41:29Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

And this is the configuration that is mentioned by the inject command:

kubectl get configmap -n istio-system istio -o yaml:

apiVersion: v1data: mesh: |- # Uncomment the following line to enable mutual TLS between proxies authPolicy: MUTUAL_TLS # # Set the following variable to true to disable policy checks by the Mixer. # Note that metrics will still be reported to the Mixer. disablePolicyChecks: false # Set enableTracing to false to disable request tracing. enableTracing: true # # To disable the mixer completely (including metrics), comment out # the following line mixerAddress: istio-mixer.istio-system:15004 # This is the ingress service name, update if you used a different name ingressService: istio-ingress # # Along with discoveryRefreshDelay, this setting determines how # frequently should Envoy fetch and update its internal configuration # from Istio Pilot. Lower refresh delay results in higher CPU # utilization and potential performance loss in exchange for faster # convergence. Tweak this value according to your setup. rdsRefreshDelay: 1s # defaultConfig: # NOTE: If you change any values in this section, make sure to make # the same changes in start up args in istio-ingress pods. # See rdsRefreshDelay for explanation about this setting. discoveryRefreshDelay: 1s # # TCP connection timeout between Envoy & the application, and between Envoys. connectTimeout: 10s # ### ADVANCED SETTINGS ############# # Where should envoy's configuration be stored in the istio-proxy container configPath: "/etc/istio/proxy" binaryPath: "/usr/local/bin/envoy" # The pseudo service name used for Envoy. serviceCluster: istio-proxy # These settings that determine how long an old Envoy # process should be kept alive after an occasional reload. drainDuration: 45s parentShutdownDuration: 1m0s # # Port where Envoy listens (on local host) for admin commands # You can exec into the istio-proxy container in a pod and # curl the admin port (curl http://localhost:15000/) to obtain # diagnostic information from Envoy. See # https://lyft.github.io/envoy/docs/operations/admin.html # for more details proxyAdminPort: 15000 # # Address where Istio Pilot service is running discoveryAddress: istio-pilot.istio-system:15003 # # Zipkin trace collector zipkinAddress: zipkin.istio-system:9411 # # Statsd metrics collector. Istio mixer exposes a UDP endpoint # to collect and convert statsd metrics into Prometheus metrics. statsdUdpAddress: istio-mixer.istio-system:9125 # Uncomment the following line to enable mutual TLS authentication between # sidecars and istio control plane. controlPlaneAuthPolicy: MUTUAL_TLSkind: ConfigMapmetadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","data":{"mesh":"# Uncomment the following line to enable mutual TLS between proxies\nauthPolicy: MUTUAL_TLS\n#\n# Set the following variable to true to disable policy checks by the Mixer.\n# Note that metrics will still be reported to the Mixer.\ndisablePolicyChecks: false\n# Set enableTracing to false to disable request tracing.\nenableTracing: true\n#\n# To disable the mixer completely (including metrics), comment out\n# the following line\nmixerAddress: istio-mixer.istio-system:15004\n# This is the ingress service name, update if you used a different name\ningressService: istio-ingress\n#\n# Along with discoveryRefreshDelay, this setting determines how\n# frequently should Envoy fetch and update its internal configuration\n# from Istio Pilot. Lower refresh delay results in higher CPU\n# utilization and potential performance loss in exchange for faster\n# convergence. Tweak this value according to your setup.\nrdsRefreshDelay: 1s\n#\ndefaultConfig:\n # NOTE: If you change any values in this section, make sure to make\n # the same changes in start up args in istio-ingress pods.\n # See rdsRefreshDelay for explanation about this setting.\n discoveryRefreshDelay: 1s\n #\n # TCP connection timeout between Envoy \u0026 the application, and between Envoys.\n connectTimeout: 10s\n #\n ### ADVANCED SETTINGS #############\n # Where should envoy's configuration be stored in the istio-proxy container\n configPath: \"/etc/istio/proxy\"\n binaryPath: \"/usr/local/bin/envoy\"\n # The pseudo service name used for Envoy.\n serviceCluster: istio-proxy\n # These settings that determine how long an old Envoy\n # process should be kept alive after an occasional reload.\n drainDuration: 45s\n parentShutdownDuration: 1m0s\n #\n # Port where Envoy listens (on local host) for admin commands\n # You can exec into the istio-proxy container in a pod and\n # curl the admin port (curl http://localhost:15000/) to obtain\n # diagnostic information from Envoy. See\n # https://lyft.github.io/envoy/docs/operations/admin.html\n # for more details\n proxyAdminPort: 15000\n #\n # Address where Istio Pilot service is running\n discoveryAddress: istio-pilot.istio-system:15003\n #\n # Zipkin trace collector\n zipkinAddress: zipkin.istio-system:9411\n #\n # Statsd metrics collector. Istio mixer exposes a UDP endpoint\n # to collect and convert statsd metrics into Prometheus metrics.\n statsdUdpAddress: istio-mixer.istio-system:9125\n # Uncomment the following line to enable mutual TLS authentication between\n # sidecars and istio control plane.\n controlPlaneAuthPolicy: MUTUAL_TLS"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"istio","namespace":"istio-system"}} creationTimestamp: 2017-11-03T16:13:30Z name: istio namespace: istio-system resourceVersion: "743" selfLink: /api/v1/namespaces/istio-system/configmaps/istio uid: ee86b9e9-c0b1-11e7-9c1f-5254009962d2

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/istio/issues/issues/117, or mute the thread https://github.com/notifications/unsubscribe-auth/AAw2d_wh0wdvdk82qHjN1m_dCS3nra-dks5sy0IqgaJpZM4QRXjb .

[ayj]

Should we consider using AllowUnknownFields for config forward compatibility? e.g.

diff --git a/pilot/model/conversion.go b/pilot/model/conversion.go
index 3d4d1869..d8f3ee88 100644
--- a/pilot/model/conversion.go
+++ b/pilot/model/conversion.go
@@ -93,7 +94,8 @@ func (ps *ProtoSchema) FromJSON(js string) (proto.Message, error) {
 // ApplyJSON unmarshals a JSON string into a proto message
 func ApplyJSON(js string, pb proto.Message) error {
-       return jsonpb.UnmarshalString(js, pb)
+       m := jsonpb.Unmarshaler{AllowUnknownFields: true}
+       return m.Unmarshal(strings.NewReader(js), pb)
 }

[kahootali]

I am also facing the same error message. I am using istio 0.2.7 and when I call

kubectl apply -f <(istioctl kube-inject -f svc.yaml)

I get

Error: could not read valid configmap "istio" from namespace "istio-system": Get <MyServerIP>/api/v1/namespaces/istio-system/configmaps/istio: x509: certificate signed by unknown authority - Re-run kube-inject with-i and ensure valid MeshConfig exists. `

Any idea?

[ayj]

@kahootali, that looks like a different issue related to mis-configured kubectl. See https://github.com/kubernetes/kubernetes/issues/53313.

[nmittler]

@ayj @mandarjog

I believe I'm running into this too. Running e2e tests locally (rebased onto the latest master), I get a proto conversion error for the field mixerCheckServer (a field in MeshConfig):

KubeInject failed command failed: "Error: failed to convert to proto. unknown field \"mixerCheckServer\" in v1alpha1.MeshConfig\n" exit status 255

I'm using HUB=gcr.io/istio-testing, TAG=330981aea07cd359234c0ae08e9512fa1e130a59 (which is from this morning)

I see the field in config.pb.go and was added around a month ago, so I'm not sure what is out-of-date exactly.

[nmittler]

@mandarjog could my issue above be related to this thread?

[mandarjog]

Yes, this means istioctl is older from before this change was made. Please use a more recent istioctl

[nmittler]

@mandarjog ah ok ... let me take a look. Thanks!

[nmittler]

@mandarjog that was indeed the problem ... thanks again!