search

particuleio / teks

Full feature EKS cluster with Terragrunt/Terraform
https://particuleio.github.io/teks/
Apache License 2.0
333 stars 80 forks source link

getting this error in nginx after deploy cert-manager and ingress tls #17

Closed ramesh-kumarjha closed 3 years ago

ramesh-kumarjha commented 3 years ago

getting error after deploy cert-manager and ingress tls and its work fine in http terragrunt.hcl file is 

 cert-manager = {
    enabled                   = true
    acme_email                = "rameshjha2022@gmail.com"
    acme_http01_enabled       = true
    acme_http01_ingress_class = "nginx"
    acme_dns01_enabled        = true
    allowed_cidrs             = local.public_subnets_cidr_blocks
    experimental_csi_driver   = true
  }
  kube-prometheus-stack = {
    enabled                     = true
    allowed_cidrs               = local.public_subnets_cidr_blocks
    thanos_sidecar_enabled      = true
    thanos_bucket_force_destroy = true
    extra_values                = <<-EXTRA_VALUES
      grafana:
        deploymentStrategy:
          type: Recreate
        ingress:
          enabled: true
          annotations:
            kubernetes.io/ingress.class: nginx
            cert-manager.io/cluster-issuer: "letsencrypt"
            kubernetes.io/tls-acme: "true"
            ingress.kubernetes.io/force-ssl-redirect: "true"    
          hosts:
            - grafana.${local.default_domain_suffix}
          tls:
            - secretName: grafana.${local.default_domain_suffix}
              hosts:
               - grafana.${local.default_domain_suffix}
        persistence:
          enabled: true
          storageClassName: ebs-sc
          accessModes:
            - ReadWriteOnce
          size: 1Gi
          }
      ------
  logs of nginx 
  ```
  "networking.k8s.io/v1beta1", ResourceVersion:"18258834", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

W0721 17:45:08.508516 6 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key "monitoring/prometheus.thanos.prom-stack.blackbucklabs.net" in local store W0721 17:45:08.510744 6 controller.go:1196] Error getting SSL certificate "monitoring/prometheus.thanos.prom-stack.blackbucklabs.net": local SSL certificate monitoring/prometheus.thanos.prom-stack.blackbucklabs.net was not found. Using default certificate

certificate is persent in nginx pod but it is taking defalut one both nginx and secrect are in same name space 

cert-manager log 
-----

kubectl logs -f cert-manager-8df74bb89-t6d4z -n cert-manager I0722 17:32:09.523111 1 start.go:74] cert-manager "msg"="starting controller" "git-commit"="614438aed00e1060870b273f2238794ef69b60ab" "version"="v1.3.1" W0722 17:32:09.523200 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0722 17:32:09.524185 1 controller.go:171] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["172.20.0.10:53"] I0722 17:32:09.524773 1 controller.go:72] cert-manager/controller "msg"="enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]" I0722 17:32:09.525467 1 controller.go:131] cert-manager/controller "msg"="starting leader election" I0722 17:32:09.526315 1 metrics.go:166] cert-manager/controller/build-context/metrics "msg"="listening for connections on" "address"={"IP":"::","Port":9402,"Zone":""} I0722 17:32:09.526724 1 leaderelection.go:243] attempting to acquire leader lease kube-system/cert-manager-controller... I0722 17:33:27.726101 1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-controller I0722 17:33:27.728026 1 reflector.go:207] Starting reflector v1.Secret (5m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.228590 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="challenges" I0722 17:33:29.228839 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-approver" I0722 17:33:29.229009 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi" I0722 17:33:29.229135 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificates-revision-manager" I0722 17:33:29.229282 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim" I0722 17:33:29.229423 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault" I0722 17:33:29.229479 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificates-issuing" I0722 17:33:29.229519 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificates-request-manager" I0722 17:33:29.229561 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme" I0722 17:33:29.229599 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca" I0722 17:33:29.229641 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="issuers" I0722 17:33:29.229683 1 reflector.go:207] Starting reflector v1.Order (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.229848 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned" I0722 17:33:29.230078 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificates-key-manager" I0722 17:33:29.230183 1 reflector.go:207] Starting reflector v1.CertificateRequest (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.230317 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificates-metrics" I0722 17:33:29.230437 1 reflector.go:207] Starting reflector v1.Certificate (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.230568 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificates-readiness" I0722 17:33:29.230702 1 reflector.go:207] Starting reflector v1.Pod (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.230829 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="certificates-trigger" I0722 17:33:29.229434 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers" I0722 17:33:29.229351 1 reflector.go:207] Starting reflector v1beta1.Ingress (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.229391 1 controller.go:105] cert-manager/controller "msg"="starting controller" "controller"="orders" I0722 17:33:29.230084 1 reflector.go:207] Starting reflector v1.Challenge (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.230154 1 reflector.go:207] Starting reflector v1.ClusterIssuer (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.230663 1 reflector.go:207] Starting reflector v1.Secret (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.230921 1 reflector.go:207] Starting reflector v1.Service (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:33:29.230119 1 reflector.go:207] Starting reflector *v1.Issuer (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 W0722 17:33:29.273356 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress W0722 17:33:29.299606 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress I0722 17:34:16.731919 1 setup.go:90] cert-manager/controller/clusterissuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" I0722 17:34:16.738384 1 setup.go:90] cert-manager/controller/clusterissuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt" "resource_namespace"="" "resource_version"="v1" I0722 17:34:16.990767 1 setup.go:178] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" I0722 17:34:17.239203 1 setup.go:178] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt" "resource_namespace"="" "resource_version"="v1" I0722 17:34:18.771611 1 setup.go:270] cert-manager/controller/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" I0722 17:34:18.771803 1 conditions.go:95] Setting lastTransitionTime for Issuer "letsencrypt-staging" condition "Ready" to 2021-07-22 17:34:18.771785968 +0000 UTC m=+129.276599161 I0722 17:34:18.835926 1 setup.go:270] cert-manager/controller/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt" "resource_namespace"="" "resource_version"="v1" I0722 17:34:18.835965 1 conditions.go:95] Setting lastTransitionTime for Issuer "letsencrypt" condition "Ready" to 2021-07-22 17:34:18.835958833 +0000 UTC m=+129.340771996 I0722 17:34:18.934824 1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" I0722 17:34:18.957978 1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt" "resource_namespace"="" "resource_version"="v1" I0722 17:34:21.994663 1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" I0722 17:34:22.239811 1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt" "resource_namespace"="" "resource_version"="v1" W0722 17:38:52.301824 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress I0722 17:42:48.343983 1 conditions.go:182] Setting lastTransitionTime for Certificate "grafana.thanos.prom-stack.blackbucklabs.net" condition "Ready" to 2021-07-22 17:42:48.343976156 +0000 UTC m=+638.848789319


cert-manager webhook log 
-----

kubectl logs -f cert-manager-webhook-86f4bbc997-kcfwx -n cert-manager W0722 17:32:07.986393 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. W0722 17:32:07.989574 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0722 17:32:07.989798 1 webhook.go:69] cert-manager/webhook "msg"="using dynamic certificate generating using CA stored in Secret resource" "secret_name"="cert-manager-webhook-ca" "secret_namespace"="cert-manager" I0722 17:32:07.990506 1 server.go:148] cert-manager/webhook "msg"="listening for insecure healthz connections" "address"=":6080" I0722 17:32:07.990585 1 server.go:161] cert-manager/webhook "msg"="listening for secure connections" "address"=":10260" I0722 17:32:07.990614 1 server.go:187] cert-manager/webhook "msg"="registered pprof handlers" I0722 17:32:07.992240 1 reflector.go:207] Starting reflector *v1.Secret (1m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156 I0722 17:32:09.127841 1 dynamic_source.go:199] cert-manager/webhook "msg"="Updated serving TLS certificate"


ingress
------

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress Name: kube-prometheus-stack-grafana Namespace: monitoring Address: ae6a3fd83c00a490c92975527b65c33a-500584658.ap-south-1.elb.amazonaws.com Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) TLS: grafana.thanos.prom-stack.blackbucklabs.net terminates grafana.thanos.prom-stack.blackbucklabs.net Rules: Host Path Backends

grafana.thanos.prom-stack.blackbucklabs.net / kube-prometheus-stack-grafana:80 (10.32.37.79:3000) Annotations: cert-manager.io/cluster-issuer: letsencrypt ingress.kubernetes.io/force-ssl-redirect: true kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: true meta.helm.sh/release-name: kube-prometheus-stack meta.helm.sh/release-namespace: monitoring Events:

ArchiFleKs commented 3 years ago

Hi,

Could you quote the code blocks with back quote ? How do you manage DNS on AWS and your route 53 configuration ?

Please past your terragrunt.hcl and Terraform/Terragrunt version.

ramesh-kumarjha commented 3 years ago 

include {
  path = "${find_in_parent_folders()}"
}

terraform {
  source = "github.com/particuleio/terraform-kubernetes-addons.git//modules/aws?ref=main"
}

dependency "eks" {
  config_path = "../eks"

  mock_outputs = {
    cluster_id              = "cluster-name"
    cluster_oidc_issuer_url = "https://oidc.eks.eu-west-3.amazonaws.com/id/0000000000000000"
  }
}

dependency "vpc" {
  config_path = "../vpc"

  mock_outputs = {
    private_subnets_cidr_blocks = [
      "10.0.0.0/16",
      "192.168.0.0/24"
    ]
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<-EOF
    provider "aws" {
      region = "${local.aws_region}"
    }
    provider "kubectl" {
      host                   = data.aws_eks_cluster.cluster.endpoint
      cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
      token                  = data.aws_eks_cluster_auth.cluster.token
      load_config_file       = false
    }
    provider "kubernetes" {
      host                   = data.aws_eks_cluster.cluster.endpoint
      cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
      token                  = data.aws_eks_cluster_auth.cluster.token
    }
    provider "helm" {
      kubernetes {
        host                   = data.aws_eks_cluster.cluster.endpoint
        cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
        token                  = data.aws_eks_cluster_auth.cluster.token
      }
    }
    data "aws_eks_cluster" "cluster" {
      name = var.cluster-name
    }
    data "aws_eks_cluster_auth" "cluster" {
      name = var.cluster-name
    }
  EOF
}

locals {
  aws_region = yamldecode(file("${find_in_parent_folders("region_values.yaml")}"))["aws_region"]
  custom_tags = merge(
    yamldecode(file("${find_in_parent_folders("global_tags.yaml")}")),
    yamldecode(file("${find_in_parent_folders("env_tags.yaml")}"))
  )
  default_domain_name   = yamldecode(file("${find_in_parent_folders("global_values.yaml")}"))["default_domain_name"]
  default_domain_suffix = "${local.custom_tags["Env"]}.${local.custom_tags["Project"]}.${local.default_domain_name}"
}

inputs = {

  cluster-name = dependency.eks.outputs.cluster_id

  tags = merge(
    local.custom_tags
  )

  eks = {
    "cluster_oidc_issuer_url" = dependency.eks.outputs.cluster_oidc_issuer_url
  }

  aws-ebs-csi-driver = {
    enabled          = true
    is_default_class = true
  }

  aws-for-fluent-bit = {
    enabled = true
  }

  aws-load-balancer-controller = {
    enabled = true
  }

  aws-node-termination-handler = {
    enabled = false
  }

  calico = {
    enabled = true
  }

  cert-manager = {
    enabled                   = true
    acme_email                = "xxx"
    acme_http01_enabled       = true
    acme_http01_ingress_class = "nginx"
    acme_dns01_enabled        = true
    allowed_cidrs             = dependency.vpc.outputs.private_subnets_cidr_blocks
    experimental_csi_driver   = true
  }

  cluster-autoscaler = {
    enabled = true
  }

  cni-metrics-helper = {
    enabled = false
  }

  external-dns = {
    external-dns = {
      enabled = false
    },
  }

  ingress-nginx = {
    enabled       = true
    use_nlb_ip    = true
    allowed_cidrs = dependency.vpc.outputs.private_subnets_cidr_blocks
  }

  istio-operator = {
    enabled = false
  }

  karma = {
    enabled = false
  }

  keycloak = {
    enabled = false
  }

  kong = {
    enabled = false
  }

  kube-prometheus-stack = {
    enabled                     = true
    allowed_cidrs               = dependency.vpc.outputs.private_subnets_cidr_blocks
    thanos_sidecar_enabled      = true
    thanos_bucket_force_destroy = true
    extra_values                = <<-EXTRA_VALUES
      grafana:
        deploymentStrategy:
          type: Recreate
        ingress:
          enabled: true
          annotations:
            kubernetes.io/ingress.class: nginx
            cert-manager.io/cluster-issuer: "letsencrypt"
          hosts:
            - grafana.${local.default_domain_suffix}
          tls:
            - secretName: grafana.${local.default_domain_suffix}
              hosts:
                - grafana.${local.default_domain_suffix}
        persistence:
          enabled: true
          storageClassName: ebs-sc
          accessModes:
            - ReadWriteOnce
          size: 1Gi
      prometheus:
        prometheusSpec:
          replicas: 1
          retention: 2d
          retentionSize: "6GB"
          ruleSelectorNilUsesHelmValues: false
          serviceMonitorSelectorNilUsesHelmValues: false
          podMonitorSelectorNilUsesHelmValues: false
          storageSpec:
            volumeClaimTemplate:
              spec:
                storageClassName: ebs-sc
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 10Gi
      EXTRA_VALUES
  }

  loki-stack = {
    enabled              = false
    bucket_force_destroy = true
  }

  metrics-server = {
    enabled       = true
    allowed_cidrs = dependency.vpc.outputs.private_subnets_cidr_blocks
  }

  npd = {
    enabled = false
  }

  sealed-secrets = {
    enabled = false
  }

  thanos = {
    enabled              = true
    generate_ca          = true
    bucket_force_destroy = true
  }

}
ramesh-kumarjha commented 3 years ago

~ terragrunt -v terragrunt version v0.31.0

ramesh-kumarjha commented 3 years ago

@ArchiFleKs any update on this how i can fixed the issue

ramesh-kumarjha commented 3 years ago

@ArchiFleKs we have one hosted zone in Route53 and we have configure external dns entries . Traffic comes to route 53 and then to elb we are using classic load blanc and elb route traffic to target group .

Screenshot 2021-07-26 at 2 38 50 PM
ArchiFleKs commented 3 years ago

@ramesh-kumarjha you do not have external-dns configured in your terragrunt file ?

Cert-manager is using HTTP01 or DNS01 challenge ?

You should be able to debug the chain starting with:

ArchiFleKs commented 3 years ago

@ramesh-kumarjha closing this, please reopen if need