search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.95k stars 4.65k forks source link

Kops not creating any instances when creating a cluster on AWS #16704

Open oceanminded opened 3 months ago

oceanminded commented 3 months ago

/kind bug

1. What kops version are you running? The command kops version, will display this information. 1.29.2

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

3. What cloud provider are you using? aws

4. What commands did you run? What is the simplest way to reproduce this issue?

export BYTERAT_IRIS_AVAILABILITY_ZONES=us-east-2a,us-east-2b,us-east-2c
export NAME=cluster-xyz.my_domain.co
export KOPS_CLUSTER_NAME=$NAME
export KOPS_STATE_STORE=s3://cluster-xyz-state-store-0
export AWS_AVAILABILITY_ZONES=us-east-2a,us-east-2b,us-east-2c
export S3_BUCKET=cluster-xyz-state-store-0
export CONTROL_PLANE_SIZE="m5.8xlarge"
export NODE_SIZE="m5.8xlarge"

kops create cluster \
    --name cluster-xyz.my_domain.co \
    --ssh-public-key ~/.ssh/my_ssh_key.pub \
    --dns-zone DNS_ZONE_HOSTED_ZONEI_ID \
    --node-count 3 \
    --zones us-east-2a,us-east-2b,us-east-2c \
    --node-size $NODE_SIZE \
    --control-plane-size $CONTROL_PLANE_SIZE \
    --control-plane-zones us-east-2a,us-east-2b,us-east-2c \
    --discovery-store=s3://cluster-xyz-oidc-store-0/${NAME}/discovery

5. What happened after the commands executed? Cluster fails validation with timeouts to api servers and no ec2 instances are created.

6. What did you expect to happen? Cluster creation

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2024-07-28T01:13:58Z"
  generation: 1
  name: cluster-xyz.my_domain.co
spec:
  additionalPolicies:
    master: |
      [
        {
          "Effect": "Allow",
          "Action": [ "elasticloadbalancing:AddListenerCertificates", "elasticloadbalancing:RemoveListenerCertificates"],
          "Resource": ["*"]
        }
      ]
  api:
    loadBalancer:
      class: Network
      type: Public
  authorization:
    rbac: {}
  awsLoadBalancerController:
    enabled: true
  certManager:
    enabled: true
  channel: stable
  cloudProvider: aws
  configBase: s3://cluster-xyz-state-store-0/cluster-xyz.my_domain.co
  dnsZone: DNS_ZONE_HOSTED_ZONEI_ID
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2a
      name: a
      volumeSize: 200
      volumeType: io1
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2b
      name: b
      volumeSize: 200
      volumeType: io1
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2c
      name: c
      volumeSize: 200
      volumeType: io1
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2a
      name: a
      volumeSize: 200
      volumeType: io1
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2b
      name: b
      volumeSize: 200
      volumeType: io1
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2c
      name: c
      volumeSize: 200
      volumeType: io1
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
    useServiceAccountExternalPermissions: true
  kubeProxy:
    enabled: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  - ::/0
  kubernetesVersion: 1.29.6
  masterPublicName: api.cluster-xyz.my_domain.co
  networkCIDR: 172.20.0.0/16
  networking:
    cilium:
      enableNodePort: true
  nonMasqueradeCIDR: 100.64.0.0/10
  serviceAccountIssuerDiscovery:
    discoveryStore: s3://cluster-xyz-oidc-store-0/cluster-xyz.my_domain.co/discovery/cluster-xyz.my_domain.co
    enableAWSOIDCProvider: true
  sshAccess:
  - 0.0.0.0/0
  - ::/0
  subnets:
  - cidr: 172.20.0.0/18
    name: us-east-2a
    type: Public
    zone: us-east-2a
  - cidr: 172.20.64.0/18
    name: us-east-2b
    type: Public
    zone: us-east-2b
  - cidr: 172.20.128.0/18
    name: us-east-2c
    type: Public
    zone: us-east-2c
  topology:
    dns:
      type: Public

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:13:59Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: control-plane-us-east-2a
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:13:59Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: control-plane-us-east-2b
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2b

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:13:59Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: control-plane-us-east-2c
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2c

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:13:59Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: nodes-us-east-2a
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-east-2a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:14:00Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: nodes-us-east-2b
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-east-2b

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:14:00Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: nodes-us-east-2c
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-east-2c

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

hakman commented 3 months ago

Hi @oceanminded. Just to check, was there any error when running kops update cluster --yes? Could you check the ASGs that were created to see if there is any error message (like not enough quota)?

k8s-triage-robot commented 2 weeks ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale