kops-controller pod not using sts vpc endpoint

[amitpd]

/kind bug

1. What kops version are you running? The command kops version, will display this information. Version 1.20.0 (git-8ea83c6d233a15dacfcc769d4d82bea3f530cf72)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag. v1.20.8

3. What cloud provider are you using? aws

4. What commands did you run? What is the simplest way to reproduce this issue?

• vpv endpoint for sts is present
• kops create -f spec.yaml
• egress proxy is not set in spec. kops is installed with offline configurations since our vpc is not connected to IGW

5. What happened after the commands executed? Master ec2 instances are up. kops-controller pod goes into CrashLoopBackOff state with below error

E0723 09:00:52.248129       1 deleg.go:144] setup "msg"="unable to create verifier" "error"="RequestError: send request failed\ncaused by: Post \"https://sts.amazonaws.com/\": dial tcp 52.46.134.192:443: i/o timeout"

6. What did you expect to happen? kops-controller should use sts vpc endpoint sts.us-east-1.amazonaws.com

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  name: testkopsissue.k8s.local
spec:
  api:
    loadBalancer:
      class: Classic
      type: Internal
  assets:
    containerProxy: xxx.abc.com
  authorization:
    rbac: {}
  channel: https://nexus.abc.com/repository/kops-channels/stable
  cloudLabels:
    cluster_name: testkopsissue.k8s.local
  cloudProvider: aws
  configBase: s3://bucket/testkopsissue.k8s.local
  containerRuntime: docker
  docker:
    ipMasq: false
    ipTables: false
    logDriver: json-file
    logLevel: warn
    logOpt:
    - max-size=100m
    - max-file=3
    packages:
      hashAmd64: 5504d190eef37355231325c176686d51ade6e0cabe2da526d561a38d8611506f
      urlAmd64: https://nexus.abc.com/repository/kops-hosted/docker/docker-19.03.15.tgz
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: a
      volumeType: gp2
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: a
      volumeType: gp2
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeControllerManager:
    clusterCIDR: 172.21.128.0/17
  kubeDNS:
    provider: CoreDNS
  kubeProxy:
    clusterCIDR: 172.21.128.0/17
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: https://nexus.abc.com/repository/kubernetes-releases/release/v1.20.8
  masterPublicName: testkopsissue.k8s.local
  networkCIDR: 10.10.10.0/24
  networkID: vpc-xxx
  networking:
    weave:
      mtu: 8912
  nonMasqueradeCIDR: 172.21.0.0/16
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 10.10.10.0/24
    id: subnet-0837285a36dc9fca8
    name: us-east-1a
    type: Private
    zone: us-east-1a
  - cidr: 10.10.10.0/24
    id: subnet-0837285a36dc9fca8
    name: utility-us-east-1a
    type: Utility
    zone: us-east-1a
  topology:
    dns:
      type: Private
    masters: private
    nodes: private

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here. kops cluster creation is successful. kops validation is failing due to kops-controller pod issue.

9. Anything else do we need to know? N/A

[johngmyers]

The question would be how get the AWS libraries to do the right thing. Maybe #12043 is necessary? The documentation isn't all that clear.

[amitpd]

Thanks @johngmyers for taking this up. Is this change going to be backported to older versions 1.20 and 1.21 as well?

[prudhvigodithi]

Hey @johngmyers, I still see this error with kops 1.21.0, Are there any plans to release a new minor version in the 1.21 series with this fix? (Version 1.21.0 (git-a5bdc3359e544b314d5695b3ed596829313fc6e3)) E0803 02:38:28.509198 1 deleg.go:144] setup "msg"="unable to create verifier" "error"="RequestError: send request failed\ncaused by: Post \"https://sts.amazonaws.com/\": dial tcp 54.239.21.217:443: i/o timeout" Please let me know if there are any workarounds to proceed further.