search

istio / old_issues_repo

Deprecated issue-tracking repo, please post new issues or feature requests to istio/istio instead.
37 stars 9 forks source link

Ingress pod on restart loop after adding TLS #326

Open stunn3r-kloud opened 6 years ago

stunn3r-kloud commented 6 years ago

Is this a BUG or FEATURE REQUEST?: BUG

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?: YES

Bug: Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version - 0.7.1 kubectl version

Client Version: v1.8.6", Server Version: v1.9.3-gke

Is Istio Auth enabled or not ? Did you install the stable istio.yaml, istio-auth.yaml.... or if using the Helm chart please provide full command line input. Tried with both Auth and without auth

What happened: after deploying istio using "istio.yaml" or "istio-auth.yaml" I created "istio-ingress-certs" needed by "istio-ingress" deployment and to be further used with kubernetes "ingress" resource. After creating certs as I created the "ingress" with TLS, istio-ingress pod went into panic mode as it couldnt reconcile the configuration and restarted when budget got from 10 to 0. After restart it again fails to reconcile configuration and goes into restart-loop This loop breaks as soon as I remove TLS from the "ingress" resource.

Here are the istio-ingress pod logs:


2018-05-07T14:06:17.381424Z info    Epoch 0 starting
2018-05-07T14:06:17.382073Z info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingress --service-node ingress~~istio-ingress-868d5f978b-54zvj.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 -l info --v2-config-only]
[2018-05-07 14:06:17.396][66][info][main] external/envoy/source/server/server.cc:178] initializing epoch 0 (hot restart version=9.200.16384.256.options=capacity=16384, num_slots=8209 hash=228984379728933363)
[2018-05-07 14:06:17.400][66][info][config] external/envoy/source/server/configuration_impl.cc:52] loading 0 listener(s)
[2018-05-07 14:06:17.400][66][info][config] external/envoy/source/server/configuration_impl.cc:92] loading tracing configuration
[2018-05-07 14:06:17.400][66][info][config] external/envoy/source/server/configuration_impl.cc:101]   loading tracing driver: envoy.zipkin
[2018-05-07 14:06:17.400][66][info][config] external/envoy/source/server/configuration_impl.cc:119] loading stats sink configuration
[2018-05-07 14:06:17.400][66][info][main] external/envoy/source/server/server.cc:353] starting main dispatch loop
[2018-05-07 14:06:17.419][66][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:127] cm init: initializing cds
[2018-05-07 14:06:17.430][66][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:382] add/update cluster out.web-1.web-perf.svc.cluster.local|http during init
[2018-05-07 14:06:17.431][66][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:382] add/update cluster out.platform-1.platform-perf.svc.cluster.local|http during init
[2018-05-07 14:06:17.432][66][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:382] add/update cluster mixer_check_server during init
[2018-05-07 14:06:17.433][66][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:382] add/update cluster mixer_report_server during init
[2018-05-07 14:06:17.434][66][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:108] cm init: initializing secondary clusters
[2018-05-07 14:06:17.436][66][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:131] cm init: all clusters initialized
[2018-05-07 14:06:17.436][66][info][main] external/envoy/source/server/server.cc:337] all clusters initialized. initializing init manager
[2018-05-07 14:06:17.440][66][info][config] src/envoy/utils/config.cc:50] v2 mixer client config: transport {
  check_cluster: "mixer_check_server"
  report_cluster: "mixer_report_server"
}
service_configs {
  key: "istio-ingress.istio-system.svc.cluster.local"
  value {
    mixer_attributes {
      attributes {
        key: "destination.service"
        value {
          string_value: "istio-ingress.istio-system.svc.cluster.local"
        }
      }
    }
  }
}
default_destination_service: "istio-ingress.istio-system.svc.cluster.local"
mixer_attributes {
  attributes {
    key: "destination.uid"
    value {
      string_value: "kubernetes://istio-ingress-868d5f978b-54zvj.istio-system"
    }
  }
}
forward_attributes {
  attributes {
    key: "source.uid"
    value {
      string_value: "kubernetes://istio-ingress-868d5f978b-54zvj.istio-system"
    }
  }
}

[2018-05-07 14:06:17.440][66][info][upstream] external/envoy/source/server/lds_api.cc:60] lds: add/update listener 'http_0.0.0.0_80'
[2018-05-07 14:06:17.441][66][info][config] src/envoy/utils/config.cc:50] v2 mixer client config: transport {
  check_cluster: "mixer_check_server"
  report_cluster: "mixer_report_server"
}
service_configs {
  key: "istio-ingress.istio-system.svc.cluster.local"
  value {
    mixer_attributes {
      attributes {
        key: "destination.service"
        value {
          string_value: "istio-ingress.istio-system.svc.cluster.local"
        }
      }
    }
  }
}
default_destination_service: "istio-ingress.istio-system.svc.cluster.local"
mixer_attributes {
  attributes {
    key: "destination.uid"
    value {
      string_value: "kubernetes://istio-ingress-868d5f978b-54zvj.istio-system"
    }
  }
}
forward_attributes {
  attributes {
    key: "source.uid"
    value {
      string_value: "kubernetes://istio-ingress-868d5f978b-54zvj.istio-system"
    }
  }
}

[2018-05-07 14:06:17.442][66][critical][assert] external/envoy/source/common/ssl/context_impl.cc:556] assert failure: cn_index >= 0
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Aborted, suspect faulting address 0x42
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:90] Backtrace obj</lib/x86_64-linux-gnu/libc.so.6> thr<0> (use tools/stack_decode.py):
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #0 0x7fe27c1f9428
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #1 0x7fe27c1fb029
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</usr/local/bin/envoy>
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #2 0x78bbc6
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #3 0x78dfef
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #4 0x65fdfc
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #5 0x611dfc
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #6 0x64bedc
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #7 0x64d425
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #8 0x868c5d
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #9 0x86c8bf
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #10 0x870351
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #11 0x8894c5
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #12 0x88b02b
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #13 0x73926d
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #14 0x7a96ab
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #15 0x7a903d
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #16 0x7aeb31
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #17 0x7aa7d0
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #18 0x7ac95a
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #19 0x767c8e
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #20 0x767e0c
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #21 0x65f0a6
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #22 0x65da0e
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #23 0x65e22d
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #24 0x657af7
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #25 0x99d461
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #26 0x99dbbe
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #27 0x6401bd
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #28 0x55f7d0
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #29 0x420f88
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</lib/x86_64-linux-gnu/libc.so.6>
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #30 0x7fe27c1e482f
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</usr/local/bin/envoy>
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #31 0x4657e8
[2018-05-07 14:06:17.442][66][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0
2018-05-07T14:06:17.453988Z warn    Epoch 0 terminated with an error: signal: aborted (core dumped)
2018-05-07T14:06:17.454026Z warn    Aborted all epochs
2018-05-07T14:06:17.454080Z info    Epoch 0: set retry delay to 1m42.4s, budget to 0
2018-05-07T14:07:59.854312Z info    Reconciling configuration (budget 0)`
``
**What you expected to happen:**
I expected istio-ingress to have remained calm

**How to reproduce it:**
1. use "istio.yaml" or "istio-auth.yaml"
2. create "istio-ingress-certs" as mentioned in documentation
3. create namespaces and deployments for your application
4. create ingress for each namespace with TLS

**Feature Request:**
N

**Describe the feature:**
NA
ghost commented 6 years ago

I encountered same problem...

istioctl version: 0.8.0 kubectl version

Client Version: v1.10.3 Server Version: v1.10.3

I created k8s cluster by kubeadm.

PiotrSikora commented 6 years ago

The stacktrace points to an assert that checks if the provided certificate has either Common Name or Subject Alternative Name(s). It seems that the certificate you're trying to use doesn't have either, and Envoy refuses to use it (I admit that this isn't the most user friendly startup error).

What's the source of your certificate?