Unable to create cluster: "error creating AutoScalingGroup: You are not authorized to use launch template"

[pulnara]

While trying to provision my K8s cluster, I'm getting the following error:

kops update cluster --name istio-workshop.k8s.local --yes
I0426 16:00:17.797006   39155 apply_cluster.go:479] Gossip DNS: skipping DNS validation
I0426 16:00:20.290104   39155 executor.go:111] Tasks: 0 done / 84 total; 42 can run
I0426 16:00:21.575762   39155 executor.go:111] Tasks: 42 done / 84 total; 17 can run
I0426 16:00:22.278848   39155 executor.go:111] Tasks: 59 done / 84 total; 19 can run
I0426 16:00:23.320090   39155 executor.go:111] Tasks: 78 done / 84 total; 2 can run
I0426 16:00:23.322165   39155 executor.go:111] Tasks: 80 done / 84 total; 2 can run
I0426 16:00:24.576413   39155 executor.go:111] Tasks: 82 done / 84 total; 2 can run
W0426 16:00:26.799408   39155 executor.go:139] error running task "AutoscalingGroup/nodes-us-east-1a.istio-workshop.k8s.local" (9m57s remaining to succeed): error creating AutoScalingGroup: You are not authorized to use launch template: lt-077576701956073c2
W0426 16:00:26.799449   39155 executor.go:139] error running task "AutoscalingGroup/master-us-east-1a.masters.istio-workshop.k8s.local" (9m57s remaining to succeed): error creating AutoScalingGroup: You are not authorized to use launch template: lt-056f220b3c6d6f7e9
I0426 16:00:26.799481   39155 executor.go:155] No progress made, sleeping before retrying 2 task(s)
I0426 16:00:36.800380   39155 executor.go:111] Tasks: 82 done / 84 total; 2 can run
W0426 16:00:38.565497   39155 executor.go:139] error running task "AutoscalingGroup/master-us-east-1a.masters.istio-workshop.k8s.local" (9m46s remaining to succeed): error creating AutoScalingGroup: You are not authorized to use launch template: lt-056f220b3c6d6f7e9
W0426 16:00:38.565528   39155 executor.go:139] error running task "AutoscalingGroup/nodes-us-east-1a.istio-workshop.k8s.local" (9m46s remaining to succeed): error creating AutoScalingGroup: You are not authorized to use launch template: lt-077576701956073c2
I0426 16:00:38.565542   39155 executor.go:155] No progress made, sleeping before retrying 2 task(s)
I0426 16:00:48.570412   39155 executor.go:111] Tasks: 82 done / 84 total; 2 can run
W0426 16:00:51.039206   39155 executor.go:139] error running task "AutoscalingGroup/nodes-us-east-1a.istio-workshop.k8s.local" (9m33s remaining to succeed): error creating AutoScalingGroup: You are not authorized to use launch template: lt-077576701956073c2
W0426 16:00:51.039244   39155 executor.go:139] error running task "AutoscalingGroup/master-us-east-1a.masters.istio-workshop.k8s.local" (9m33s remaining to succeed): error creating AutoScalingGroup: You are not authorized to use launch template: lt-056f220b3c6d6f7e9
I0426 16:00:51.039601   39155 executor.go:155] No progress made, sleeping before retrying 2 task(s)
I0426 16:01:01.040001   39155 executor.go:111] Tasks: 82 done / 84 total; 2 can run

I have correct instance type set in my Launch Templates, and most recent credentials in the ~/.aws/credentials file:

[bzurkowski]

What version of kops are you using?

$ kops version

What command did you use to create the cluster config (kops create ...)?

Also, please paste the cluster configuration:

$ kops edit cluster $NAME

[pulnara]

I'm using kops 1.20.0. I tried installing version 1.18.3 mentioned in the tutorial, but kops refused to work, telling me to upgrade to newer version:

$ kops update cluster --name istio-workshop.k8s.local --yes 

*********************************************************************************

A new kops version is available: 1.20.0
Upgrading is recommended
More information: https://github.com/kubernetes/kops/blob/master/permalinks/upgrade_kops.md#1.20.0

*********************************************************************************

*********************************************************************************

This version of kubernetes is not yet supported; upgrading kops is required
(you can bypass this check by exporting KOPS_RUN_TOO_NEW_VERSION)

*********************************************************************************

I'll try downgrading both kops and kubectl (to v1.18.1 mentioned in the tutorial).

My current kops version:

$ kops version
Version 1.20.0 (git-8ea83c6d233a15dacfcc769d4d82bea3f530cf72)

And kubectl:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"darwin/amd64"}

The command I used to create the cluster config:

$ kops create cluster \
>     --zones=us-east-1a \
>     --master-count=1 \
>     --node-count=2 \
>     --node-size=t3.medium \
>     --master-size=t3.medium \
>     --networking=calico \
>     ${NAME}

After that, I navigated to my Launch Templates, changed instance type and default template version.

Using an other command:

$ kops create cluster --name ${NAME} --zones us-east-1a --state ${KOPS_STATE_STORE} --yes

Doesn't work either.

My cluster config:

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2021-04-26T15:09:58Z"
  name: istio-workshop.k8s.local
spec:
  api:
    loadBalancer:
      class: Classic
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://istio-workshop-11902-kops-cluster-state-store/istio-workshop.k8s.local
  containerRuntime: containerd
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: a
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: a
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.20.6
  masterInternalName: api.internal.istio-workshop.k8s.local
  masterPublicName: api.istio-workshop.k8s.local
  networkCIDR: 172.20.0.0/16
  networking:
    calico: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.20.32.0/19
    name: us-east-1a
    type: Public
    zone: us-east-1a
  topology:
    dns:
      type: Public
    masters: public
    nodes: public

[pulnara]

Downgrading both kubectl and kops version didn't help.

[bzurkowski]

@pulnara First off, ensure that your AWS client works properly. Try to execute the following command:

$ aws ec2 describe-instances

It should respond with an empty table or a list of running instances. Please, provide its output.

Remember that the AWS session expires after 3 hours and the credentials must be updated from the Vocareum dashboard.

[pulnara]

I believe it does:

$ aws ec2 describe-instances
-------------------
|DescribeInstances|
+-----------------+

I tried updating the credentials, but it didn't help.

[bzurkowski]

I see. Then, let's try to redeploy the cluster.

First, ensure the old cluster is destroyed. Use commands from this chapter. It might fail in which case please provide the command output.

Next, ensure there are no running EC2 instances in AWS. According to https://github.com/istioworkshop/istio-workshop/issues/8#issuecomment-826953469 there are none.

Finally, restart your terminal and try to provision the cluster using the approach from the previous lab - ignore instructions from the Provision a cluster section. Name the cluster with a different name than istio-workshop.k8s.local.

[bzurkowski]

Possibly related to https://github.com/istioworkshop/istio-workshop/issues/9#issuecomment-827122522.