Cannot create cluster due to oservice-account-jwks-uri setting

[pawel-powroznik]

1. What kops version are you running? The command kops version, will display this information. ❯ kops version Version 1.20.1 (git-5a27dad40a703f646433595a2a40cf94a0c43cd5)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

ubuntu@ip-172-20-38-9:/etc/kubernetes/manifests$ kubectl version Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:40:09Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:32:49Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

image: k8s.gcr.io/kube-apiserver:v1.20.7
image: k8s.gcr.io/kube-controller-manager:v1.20.7
image: k8s.gcr.io/kube-proxy:v1.20.7
image: k8s.gcr.io/kube-scheduler:v1.20.7
kubernetesVersion: 1.20.7

3. What cloud provider are you using? AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

create new k8s cluster by running:

  kops create cluster -v 10 \
  --zones us-west-2a \
  --name ${NAME} \
  --kubernetes-version $VERSION \
  --authorization RBAC \
  --ssh-public-key <CERT> \
  --admin-access $my_ip/32 \
  --node-size m5.4xlarge \
  --master-size t3.2xlarge \
  --node-count 4 \
  --networking calico \
  --bastion \
  --topology private \

Add the config for ServiceAccountTokenVolumeProjection feature in Apiiserver

spec:
  metricsServer:
    enabled: true
    insecure: true
  kubeAPIServer:
    apiAudiences:
    - api
    - istio-ca
    featureGates:
      TokenRequest: "true"
    serviceAccountIssuer: kubernetes.default.svc
    serviceAccountKeyFile:
    - /srv/kubernetes/server.key
    serviceAccountSigningKeyFile: /srv/kubernetes/server.key

5. What happened after the commands executed? API server container could not start. K8s cluster not available.

After checking crio logs:

  ubuntu@ip-172-20-38-9:/etc/kubernetes/manifests$ sudo crictl logs 87f1cd71353b7
  ...
  I0528 15:45:59.235757       1 flags.go:59] FLAG: --service-account-api-audiences="[api,istio-ca]"
  I0528 15:45:59.235762       1 flags.go:59] FLAG: --service-account-extend-token-expiration="true"
  I0528 15:45:59.235766       1 flags.go:59] FLAG: --service-account-issuer="kubernetes.default.svc"
  I0528 15:45:59.235769       1 flags.go:59] FLAG: --service-account-jwks-uri="kubernetes.default.svc/openid/v1/jwks"
  I0528 15:45:59.235773       1 flags.go:59] FLAG: --service-account-key-file="[/srv/kubernetes/server.key,/srv/kubernetes/service-account.key]"
  I0528 15:45:59.235778       1 flags.go:59] FLAG: --service-account-lookup="true"
  I0528 15:45:59.235781       1 flags.go:59] FLAG: --service-account-max-token-expiration="0s"
  I0528 15:45:59.235784       1 flags.go:59] FLAG: --service-account-signing-key-file="/srv/kubernetes/server.key"
  I0528 15:45:59.235788       1 flags.go:59] FLAG: --service-cluster-ip-range="100.64.0.0/13"
  I0528 15:45:59.235791       1 flags.go:59] FLAG: --service-node-port-range="30000-32767"
  I0528 15:45:59.235797       1 flags.go:59] FLAG: --show-hidden-metrics-for-version=""
  I0528 15:45:59.235800       1 flags.go:59] FLAG: --shutdown-delay-duration="0s"
  I0528 15:45:59.235802       1 flags.go:59] FLAG: --skip-headers="false"
  I0528 15:45:59.235806       1 flags.go:59] FLAG: --skip-log-headers="false"
  I0528 15:45:59.235809       1 flags.go:59] FLAG: --ssh-keyfile=""
  I0528 15:45:59.235812       1 flags.go:59] FLAG: --ssh-user=""
  I0528 15:45:59.235814       1 flags.go:59] FLAG: --stderrthreshold="2"
  I0528 15:45:59.235818       1 flags.go:59] FLAG: --storage-backend="etcd3"
  I0528 15:45:59.235821       1 flags.go:59] FLAG: --storage-media-type="application/vnd.kubernetes.protobuf"
  I0528 15:45:59.235825       1 flags.go:59] FLAG: --target-ram-mb="0"
  I0528 15:45:59.235828       1 flags.go:59] FLAG: --tls-cert-file="/srv/kubernetes/server.crt"
  I0528 15:45:59.235831       1 flags.go:59] FLAG: --tls-cipher-suites="[]"
  I0528 15:45:59.235837       1 flags.go:59] FLAG: --tls-min-version=""
  I0528 15:45:59.235840       1 flags.go:59] FLAG: --tls-private-key-file="/srv/kubernetes/server.key"
  I0528 15:45:59.235843       1 flags.go:59] FLAG: --tls-sni-cert-key="[]"
  I0528 15:45:59.235848       1 flags.go:59] FLAG: --token-auth-file=""
  I0528 15:45:59.235851       1 flags.go:59] FLAG: --v="2"
  I0528 15:45:59.235854       1 flags.go:59] FLAG: --version="false"
  I0528 15:45:59.235859       1 flags.go:59] FLAG: --vmodule=""
  I0528 15:45:59.235863       1 flags.go:59] FLAG: --watch-cache="true"
  I0528 15:45:59.235866       1 flags.go:59] FLAG: --watch-cache-sizes="[]"
  I0528 15:45:59.236281       1 server.go:632] external host was not specified, using 172.20.38.9
  Error: service-account-jwks-uri requires https scheme, parsed as: kubernetes.default.svc/openid/v1/jwks

6. What did you expect to happen?

Apiserver should be up and running, cluster should be available

**7. Please provide your cluster manifest. Execute

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2021-05-28T15:02:39Z"
  generation: 1
  name: <NAME>
spec:
  api:
    loadBalancer:
      class: Classic
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://<SNIPPED>
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-west-2a
      name: a
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-west-2a
      name: a
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    apiAudiences:
    - api
    - istio-ca
    featureGates:
      TokenRequest: "true"
    serviceAccountIssuer: kubernetes.default.svc
    serviceAccountKeyFile:
    - /srv/kubernetes/server.key
    serviceAccountSigningKeyFile: /srv/kubernetes/server.key
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - <SNIPPED>/32
  kubernetesVersion: v1.20.7
  masterInternalName: <SNIPPED>
  masterPublicName: <SNIPPED>
  metricsServer:
    enabled: true
    insecure: true
  networkCIDR: 172.20.0.0/16
  networking:
    calico: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - <SNIPPED>/32
  subnets:
  - cidr: 172.20.32.0/19
    name: us-west-2a
    type: Private
    zone: us-west-2a
  - cidr: 172.20.0.0/22
    name: utility-us-west-2a
    type: Utility
    zone: us-west-2a
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

The workaround: Modify kube-apiserver yaml manifest:

vim /etc/kubernetes/manifests/kube-apiserver.manifest

Remove the following line:

    - --service-account-jwks-uri=kubernetes.default.svc/openid/v1/jwks

Container should start then.

[johngmyers]

kOps no longer uses /srv/kubernetes/server.key as the service account signing key. The key in that file is now different for each control plane node. Why are you overriding those settings like that?

[pawel-powroznik]

I used this key for ServiceAccountTokenVolumeProjection feature which is required to install Istio. https://github.com/istio/istio/issues/17378#issuecomment-594366491

This was working with k8s 1.18 and 1.19.

[johngmyers]

That information is out of date. See https://kops.sigs.k8s.io/operations/service_account_token_volumes

[olemarkus]

I think those docs may also be a little bit out of date.

kOps 1.20+ will set the issuer settings correctly by default. When istio uses ServiceAccountTokenVolumeProjection, I assume it sets the correct audience on the projected token directly as well. So removing those API server settings will most likely make things work properly.

[pawel-powroznik]

I can confirm that kops 1.20.1 sets Service Account Token Volume Projection by default, without any custom kubeapi settings. This issue can be closed. Probably below docs need to be updated: https://kops.sigs.k8s.io/operations/service_account_token_volumes/

[olemarkus]

Just to confirm, you are not setting any custom audience either (istio-ca), right?

[pawel-powroznik]

yes, I do not have any custom kubeapi settings. Only defaults created by running below command:

  kops create cluster -v 10 \
  --zones us-west-2a \
  --name ${NAME} \
  --kubernetes-version $VERSION \
  --authorization RBAC \
  --ssh-public-key <CERT> \
  --admin-access $my_ip/32 \
  --node-size m5.4xlarge \
  --master-size t3.2xlarge \
  --node-count 4 \
  --networking calico \
  --bastion \
  --topology private \