search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.87k stars 4.64k forks source link

error getting ingress status w/ terraform output #6788

Closed nderraugh closed 3 years ago

nderraugh commented 5 years ago

1. What kops version are you running? The command kops version, will display this information.

Version 1.12.0-beta.2 (git-d1453d22a)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

--kubernetes-version 1.12.7

3. What cloud provider are you using?

GCE

4. What commands did you run? What is the simplest way to reproduce this issue?

kops create cluster --name $CLUSTER_NAME.k8s.local --zones us-west1-b \
  --state $KOPS_STATE_STORE --project=$PROJECT \
  --kubernetes-version 1.12.7 --out=./tf-out \
  --target=terraform

5. What happened after the commands executed?

error getting ingress status: error getting ForwardingRule "api-haughty-seniority-k8s-local": googleapi: Error 404: The resource 'projects/kops-orkestra/regions/us-west1/forwardingRules/api-haughty-seniority-k8s-local' was not found, notFound

6. What did you expect to happen?

Error-free output of terraform to ./tf-out

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.


apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2019-04-17T14:24:36Z
  name: haughty-seniority.k8s.local
spec:
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: gce
  configBase: gs://kubernetes-clusters-yopp-kops-devops/haughty-seniority.k8s.local
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-us-west1-b
      name: b
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-us-west1-b
      name: b
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.12.7
  masterPublicName: api.haughty-seniority.k8s.local
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  project: kops-orkestra
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - name: us-west1
    region: us-west1
    type: Public
  topology:
    dns:
      type: Public
    masters: public
    nodes: public

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2019-04-17T14:24:36Z
  labels:
    kops.k8s.io/cluster: haughty-seniority.k8s.local
  name: master-us-west1-b
spec:
  image: cos-cloud/cos-stable-65-10323-99-0
  machineType: n1-standard-1
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-us-west1-b
  role: Master
  subnets:
  - us-west1
  zones:
  - us-west1-b

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2019-04-17T14:24:37Z
  labels:
    kops.k8s.io/cluster: haughty-seniority.k8s.local
  name: nodes
spec:
  image: cos-cloud/cos-stable-65-10323-99-0
  machineType: n1-standard-2
  maxSize: 2
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - us-west1
  zones:
  - us-west1-b

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

Gist

9. Anything else do we need to know?

There are no errors if I omit the terraform options. i.e. 

--out=./tf-out --target=terraform
fejta-bot commented 5 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 5 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 5 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 5 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/kops/issues/6788#issuecomment-531495523): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
MoSheikh commented 3 years ago

I'm having the same issue when initializing a new cluster on GCE.

rifelpet commented 3 years ago

/reopen

can you post your cluster spec and the generated terraform code?

k8s-ci-robot commented 3 years ago

@rifelpet: Reopened this issue.

In response to [this](https://github.com/kubernetes/kops/issues/6788#issuecomment-786411525): >/reopen > >can you post your cluster spec and the generated terraform code? Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
MoSheikh commented 3 years ago

@rifelpet

Sorry for the delay.

Looks like the error doesn't occur after initializing the cluster with kops update -y (without the --target=terraform) first, waiting for the resources to be deployed on cloud, and then running kops update -y --out=. --target=terraform after validating the cluster.

Here's the cluster spec:

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2021-02-27T00:34:31Z"
  name: api.dev.k8s.local
spec:
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudConfig:
    gceServiceAccount: master-ue1@example-302702.iam.gserviceaccount.com
  cloudProvider: gce
  configBase: gs://devops.example.app/kops/state/api.dev.k8s.local
  containerRuntime: containerd
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-us-east1-b
      name: b
      volumeType: pd-standard
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-us-east1-b
      name: b
      volumeType: pd-standard
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.20.2
  masterPublicName: api.api.dev.k8s.local
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  project: example-302702
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - name: us-east1
    region: us-east1
    type: Public
  topology:
    dns:
      type: Public
    masters: public
    nodes: public

And here's the generated kubernetes.tf:

locals {
  cluster_name = "api.dev.k8s.local"
  project      = "example-302702"
  region       = "us-east1"
}

output "cluster_name" {
  value = "api.dev.k8s.local"
}

output "project" {
  value = "example-302702"
}

output "region" {
  value = "us-east1"
}

provider "google" {
  region = "us-east1"
}

resource "google_compute_address" "api-api-dev-k8s-local" {
  name = "api-api-dev-k8s-local"
}

resource "google_compute_disk" "b-etcd-events-api-dev-k8s-local" {
  labels = {
    "k8s-io-cluster-name" = "api-dev-k8s-local"
    "k8s-io-etcd-events"  = "b-2fb"
    "k8s-io-role-master"  = "master"
  }
  name = "b-etcd-events-api-dev-k8s-local"
  size = 20
  type = "pd-standard"
  zone = "us-east1-b"
}

resource "google_compute_disk" "b-etcd-main-api-dev-k8s-local" {
  labels = {
    "k8s-io-cluster-name" = "api-dev-k8s-local"
    "k8s-io-etcd-main"    = "b-2fb"
    "k8s-io-role-master"  = "master"
  }
  name = "b-etcd-main-api-dev-k8s-local"
  size = 20
  type = "pd-standard"
  zone = "us-east1-b"
}

resource "google_compute_firewall" "cidr-to-master-api-dev-k8s-local" {
  allow {
    ports    = ["443"]
    protocol = "tcp"
  }
  allow {
    ports    = ["4194"]
    protocol = "tcp"
  }
  name          = "cidr-to-master-api-dev-k8s-local"
  network       = google_compute_network.default.name
  source_ranges = ["100.64.0.0/10"]
  target_tags   = ["api-dev-k8s-local-k8s-io-role-master"]
}

resource "google_compute_firewall" "cidr-to-node-api-dev-k8s-local" {
  allow {
    protocol = "tcp"
  }
  allow {
    protocol = "udp"
  }
  allow {
    protocol = "icmp"
  }
  allow {
    protocol = "esp"
  }
  allow {
    protocol = "ah"
  }
  allow {
    protocol = "sctp"
  }
  name          = "cidr-to-node-api-dev-k8s-local"
  network       = google_compute_network.default.name
  source_ranges = ["100.64.0.0/10"]
  target_tags   = ["api-dev-k8s-local-k8s-io-role-node"]
}

resource "google_compute_firewall" "https-api-api-dev-k8s-local" {
  allow {
    ports    = ["443"]
    protocol = "tcp"
  }
  name          = "https-api-api-dev-k8s-local"
  network       = google_compute_network.default.name
  source_ranges = ["0.0.0.0/0"]
  target_tags   = ["api-dev-k8s-local-k8s-io-role-master"]
}

resource "google_compute_firewall" "master-to-master-api-dev-k8s-local" {
  allow {
    protocol = "tcp"
  }
  allow {
    protocol = "udp"
  }
  allow {
    protocol = "icmp"
  }
  allow {
    protocol = "esp"
  }
  allow {
    protocol = "ah"
  }
  allow {
    protocol = "sctp"
  }
  name        = "master-to-master-api-dev-k8s-local"
  network     = google_compute_network.default.name
  source_tags = ["api-dev-k8s-local-k8s-io-role-master"]
  target_tags = ["api-dev-k8s-local-k8s-io-role-master"]
}

resource "google_compute_firewall" "master-to-node-api-dev-k8s-local" {
  allow {
    protocol = "tcp"
  }
  allow {
    protocol = "udp"
  }
  allow {
    protocol = "icmp"
  }
  allow {
    protocol = "esp"
  }
  allow {
    protocol = "ah"
  }
  allow {
    protocol = "sctp"
  }
  name        = "master-to-node-api-dev-k8s-local"
  network     = google_compute_network.default.name
  source_tags = ["api-dev-k8s-local-k8s-io-role-master"]
  target_tags = ["api-dev-k8s-local-k8s-io-role-node"]
}

resource "google_compute_firewall" "node-to-master-api-dev-k8s-local" {
  allow {
    ports    = ["443"]
    protocol = "tcp"
  }
  allow {
    ports    = ["4194"]
    protocol = "tcp"
  }
  name        = "node-to-master-api-dev-k8s-local"
  network     = google_compute_network.default.name
  source_tags = ["api-dev-k8s-local-k8s-io-role-node"]
  target_tags = ["api-dev-k8s-local-k8s-io-role-master"]
}

resource "google_compute_firewall" "node-to-node-api-dev-k8s-local" {
  allow {
    protocol = "tcp"
  }
  allow {
    protocol = "udp"
  }
  allow {
    protocol = "icmp"
  }
  allow {
    protocol = "esp"
  }
  allow {
    protocol = "ah"
  }
  allow {
    protocol = "sctp"
  }
  name        = "node-to-node-api-dev-k8s-local"
  network     = google_compute_network.default.name
  source_tags = ["api-dev-k8s-local-k8s-io-role-node"]
  target_tags = ["api-dev-k8s-local-k8s-io-role-node"]
}

resource "google_compute_firewall" "nodeport-external-to-node-api-dev-k8s-local" {
  allow {
    ports    = ["30000-32767"]
    protocol = "tcp"
  }
  allow {
    ports    = ["30000-32767"]
    protocol = "udp"
  }
  name        = "nodeport-external-to-node-api-dev-k8s-local"
  network     = google_compute_network.default.name
  source_tags = ["api-dev-k8s-local-k8s-io-role-node"]
  target_tags = ["api-dev-k8s-local-k8s-io-role-node"]
}

resource "google_compute_firewall" "ssh-external-to-master-api-dev-k8s-local" {
  allow {
    ports    = ["22"]
    protocol = "tcp"
  }
  name          = "ssh-external-to-master-api-dev-k8s-local"
  network       = google_compute_network.default.name
  source_ranges = ["0.0.0.0/0"]
  target_tags   = ["api-dev-k8s-local-k8s-io-role-master"]
}

resource "google_compute_firewall" "ssh-external-to-node-api-dev-k8s-local" {
  allow {
    ports    = ["22"]
    protocol = "tcp"
  }
  name          = "ssh-external-to-node-api-dev-k8s-local"
  network       = google_compute_network.default.name
  source_ranges = ["0.0.0.0/0"]
  target_tags   = ["api-dev-k8s-local-k8s-io-role-node"]
}

resource "google_compute_forwarding_rule" "api-api-dev-k8s-local" {
  ip_address  = google_compute_address.api-api-dev-k8s-local.address
  ip_protocol = "TCP"
  name        = "api-api-dev-k8s-local"
  port_range  = "443-443"
  target      = google_compute_target_pool.api-api-dev-k8s-local.self_link
}

resource "google_compute_instance_group_manager" "b-master-us-east1-b-api-dev-k8s-local" {
  base_instance_name = "master-us-east1-b"
  name               = "b-master-us-east1-b-api-dev-k8s-local"
  target_pools       = [google_compute_target_pool.api-api-dev-k8s-local.self_link]
  target_size        = 1
  version {
    instance_template = google_compute_instance_template.master-us-east1-b-api-dev-k8s-local.self_link
  }
  zone = "us-east1-b"
}

resource "google_compute_instance_group_manager" "b-nodes-us-east1-b-api-dev-k8s-local" {
  base_instance_name = "nodes-us-east1-b"
  name               = "b-nodes-us-east1-b-api-dev-k8s-local"
  target_size        = 1
  version {
    instance_template = google_compute_instance_template.nodes-us-east1-b-api-dev-k8s-local.self_link
  }
  zone = "us-east1-b"
}

resource "google_compute_instance_template" "master-us-east1-b-api-dev-k8s-local" {
  can_ip_forward = true
  disk {
    auto_delete  = true
    boot         = true
    device_name  = "persistent-disks-0"
    disk_name    = ""
    disk_size_gb = 20
    disk_type    = "pd-standard"
    interface    = ""
    mode         = "READ_WRITE"
    source       = ""
    source_image = "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-10-buster-v20210122"
    type         = "PERSISTENT"
  }
  machine_type = "e2-medium"
  metadata = {
    "cluster-name"                    = "api.dev.k8s.local"
    "kops-k8s-io-instance-group-name" = "master-us-east1-b"
    "startup-script"                  = file("${path.module}/data/google_compute_instance_template_master-us-east1-b-api-dev-k8s-local_metadata_startup-script")
  }
  name_prefix = "master-us-east1-b-api-dev-1v1r3d-"
  network_interface {
    access_config {
    }
    network = google_compute_network.default.name
  }
  scheduling {
    automatic_restart   = true
    on_host_maintenance = "MIGRATE"
    preemptible         = false
  }
  service_account {
    email  = "master-ue1@example-302702.iam.gserviceaccount.com"
    scopes = ["https://www.googleapis.com/auth/compute", "https://www.googleapis.com/auth/monitoring", "https://www.googleapis.com/auth/logging.write", "https://www.googleapis.com/auth/devstorage.read_write", "https://www.googleapis.com/auth/ndev.clouddns.readwrite"]
  }
  tags = ["api-dev-k8s-local-k8s-io-role-master"]
}

resource "google_compute_instance_template" "nodes-us-east1-b-api-dev-k8s-local" {
  can_ip_forward = true
  disk {
    auto_delete  = true
    boot         = true
    device_name  = "persistent-disks-0"
    disk_name    = ""
    disk_size_gb = 20
    disk_type    = "pd-standard"
    interface    = ""
    mode         = "READ_WRITE"
    source       = ""
    source_image = "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-10-buster-v20210122"
    type         = "PERSISTENT"
  }
  machine_type = "e2-medium"
  metadata = {
    "cluster-name"                    = "api.dev.k8s.local"
    "kops-k8s-io-instance-group-name" = "nodes-us-east1-b"
    "startup-script"                  = file("${path.module}/data/google_compute_instance_template_nodes-us-east1-b-api-dev-k8s-local_metadata_startup-script")
  }
  name_prefix = "nodes-us-east1-b-api-dev--upfv41-"
  network_interface {
    access_config {
    }
    network = google_compute_network.default.name
  }
  scheduling {
    automatic_restart   = true
    on_host_maintenance = "MIGRATE"
    preemptible         = false
  }
  service_account {
    email  = "master-ue1@example-302702.iam.gserviceaccount.com"
    scopes = ["https://www.googleapis.com/auth/compute", "https://www.googleapis.com/auth/monitoring", "https://www.googleapis.com/auth/logging.write", "https://www.googleapis.com/auth/devstorage.read_only"]
  }
  tags = ["api-dev-k8s-local-k8s-io-role-node"]
}

resource "google_compute_network" "default" {
  auto_create_subnetworks = true
  name                    = "default"
}

resource "google_compute_target_pool" "api-api-dev-k8s-local" {
  description      = ""
  name             = "api-api-dev-k8s-local"
  session_affinity = ""
}

terraform {
  required_version = ">= 0.12.26"
  required_providers {
    google = {
      "source"  = "hashicorp/google"
      "version" = ">= 2.19.0"
    }
  }
}
MoSheikh commented 3 years ago

It looks like this is an issue with gossip-based clusters. Trying a setup with --dns-zone set runs everything fine.

fejta-bot commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community. /close

k8s-ci-robot commented 3 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/kops/issues/6788#issuecomment-809034499): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.