search

knative / serving

Kubernetes-based, scale-to-zero, request-driven compute
https://knative.dev/docs/serving/
Apache License 2.0
5.54k stars 1.15k forks source link

AKS pod activator CrashLoopBackOff #4081

Closed magicLian closed 5 years ago

magicLian commented 5 years ago

I use the docs command

kubectl apply --selector knative.dev/crd-install=true \ --filename https://github.com/knative/serving/releases/download/v0.5.2/serving.yaml \ --filename https://github.com/knative/build/releases/download/v0.5.0/build.yaml \ --filename https://github.com/knative/eventing/releases/download/v0.5.0/release.yaml \ --filename https://github.com/knative/eventing-sources/releases/download/v0.5.0/eventing-sources.yaml \ --filename https://github.com/knative/serving/releases/download/v0.5.2/monitoring.yaml \ --filename https://raw.githubusercontent.com/knative/serving/v0.5.2/third_party/config/build/clusterrole.yaml

and the

kubectl apply --filename https://github.com/knative/serving/releases/download/v0.5.2/serving.yaml \ --filename https://github.com/knative/build/releases/download/v0.5.0/build.yaml \ --filename https://github.com/knative/eventing/releases/download/v0.5.0/release.yaml \ --filename https://github.com/knative/eventing-sources/releases/download/v0.5.0/eventing-sources.yaml \ --filename https://github.com/knative/serving/releases/download/v0.5.2/monitoring.yaml \ --filename https://raw.githubusercontent.com/knative/serving/v0.5.2/third_party/config/build/clusterrole.yaml

The result is :

NAME READY STATUS RESTARTS AGE activator-55c4768746-5bpxk 1/2 CrashLoopBackOff 45 131m autoscaler-75bdf5d8b-7cg75 2/2 Running 0 131m controller-7fbb4448f6-wd5dl 1/1 Running 0 131m webhook-54cc969f79-8b4d9 1/1 Running 0 131m

NAME READY STATUS RESTARTS AGE build-controller-694d8444f8-q4kxb 1/1 Running 0 131m build-webhook-7d9b46cdd7-86rnv 1/1 Running 0 131m

NAME READY STATUS RESTARTS AGE eventing-controller-774f79f989-ltcl4 1/1 Running 0 131m in-memory-channel-controller-5c686c86c7-ldbs4 1/1 Running 0 131m in-memory-channel-dispatcher-7bcd7f556-dqgb6 1/2 CrashLoopBackOff 30 131m webhook-5b689bfcc4-tn9jz 1/1 Running 0 131m

NAME READY STATUS RESTARTS AGE controller-manager-0 1/1 Running 0 131m

NAME READY STATUS RESTARTS AGE elasticsearch-logging-0 1/1 Running 0 131m elasticsearch-logging-1 1/1 Running 0 131m grafana-7c5b4595b8-6bx6k 1/1 Running 0 130m kibana-logging-5d8bdbdbf7-klvqx 1/1 Running 0 131m kube-state-metrics-7b58689cf9-6j9nt 4/4 Running 0 130m node-exporter-5r9xn 2/2 Running 0 130m node-exporter-kk775 2/2 Running 0 130m node-exporter-qmvr2 2/2 Running 0 130m node-exporter-vmhdb 2/2 Running 0 130m node-exporter-wkbhz 2/2 Running 0 130m node-exporter-xlpqt 0/2 Pending 0 130m prometheus-system-0 1/1 Running 0 130m prometheus-system-1 1/1 Running 0 130m

The serving ns pod activator CrashLoopBackOff and it restarts for 45 times. describe is : ` root@master:/home/master/dockerFile/knative-demo# kubectl describe pod -n knative-serving activator-55c4768746-5bpxk Name: activator-55c4768746-5bpxk Namespace: knative-serving Priority: 0 PriorityClassName: Node: aks-agentpool-40400703-3/10.240.0.128 Start Time: Mon, 13 May 2019 14:19:23 +0800 Labels: app=activator pod-template-hash=55c4768746 role=activator serving.knative.dev/release=devel Annotations: sidecar.istio.io/inject: true sidecar.istio.io/status: {"version":"c3e1cae4ba6edc90052026dd7913ae40955b8500a82aae7245ab0d1059f37e54","initContainers":["istio-init"],"containers":["istio-proxy"]... Status: Running IP: 10.240.0.150 Controlled By: ReplicaSet/activator-55c4768746 Init Containers: istio-init: Container ID: docker://d5741497dec92a0e6b536421edc7b7e85822f71d772537890bf425d2d3fc648f Image: docker.io/istio/proxy_init:1.0.7 Image ID: docker-pullable://istio/proxy_init@sha256:fa41549250eef900072bd96bbb9f45c8b35ae81465bfbc06c317e256d74a064d Port: Host Port: Args: -p 15001 -u 1337 -m REDIRECT -i * -x

  -b
  8080,8081,9090
  -d

State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Mon, 13 May 2019 14:19:52 +0800
  Finished:     Mon, 13 May 2019 14:19:53 +0800
Ready:          True
Restart Count:  0
Environment:    <none>
Mounts:         <none>

Containers: activator: Container ID: docker://f281f3bdd43f9aa609183fd67134c9093e40ad721e3f6d75e55ef0ffe4bfa029 Image: gcr.io/knative-releases/github.com/knative/serving/cmd/activator@sha256:c75dc977b2a4d16f01f89a1741d6895990b7404b03ffb45725a63104d267b74a Image ID: docker-pullable://gcr.io/knative-releases/github.com/knative/serving/cmd/activator@sha256:c75dc977b2a4d16f01f89a1741d6895990b7404b03ffb45725a63104d267b74a Ports: 8080/TCP, 8081/TCP, 9090/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Args: -logtostderr=false -stderrthreshold=FATAL State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Mon, 13 May 2019 16:25:33 +0800 Finished: Mon, 13 May 2019 16:26:01 +0800 Ready: False Restart Count: 45 Limits: cpu: 200m memory: 600Mi Requests: cpu: 20m memory: 60Mi Liveness: http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: activator-55c4768746-5bpxk (v1:metadata.name) SYSTEM_NAMESPACE: knative-serving (v1:metadata.namespace) CONFIG_LOGGING_NAME: config-logging KUBERNETES_PORT_443_TCP_ADDR: aksstage-dc3ad954.hcp.eastasia.azmk8s.io KUBERNETES_PORT: tcp://aksstage-dc3ad954.hcp.eastasia.azmk8s.io:443 KUBERNETES_PORT_443_TCP: tcp://aksstage-dc3ad954.hcp.eastasia.azmk8s.io:443 KUBERNETES_SERVICE_HOST: aksstage-dc3ad954.hcp.eastasia.azmk8s.io Mounts: /etc/config-logging from config-logging (rw) /etc/config-observability from config-observability (rw) /var/run/secrets/kubernetes.io/serviceaccount from controller-token-rkmfr (ro) istio-proxy: Container ID: docker://2a717ad48b66faad0712c61863efd39dfb5a6f1c52ebfbb40278a8f82160b408 Image: docker.io/istio/proxyv2:1.0.7 Image ID: docker-pullable://istio/proxyv2@sha256:9c6663cddbc984e88c27530d8acac7dca83070c4ad6d2570604cc4fff6c36a7a Port: 15090/TCP Host Port: 0/TCP Args: proxy sidecar --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster activator --drainDuration 45s --parentShutdownDuration 1m0s --discoveryAddress istio-pilot.istio-system:15007 --discoveryRefreshDelay 1s --zipkinAddress zipkin.istio-system:9411 --connectTimeout 10s --proxyAdminPort 15000 --controlPlaneAuthPolicy NONE State: Running Started: Mon, 13 May 2019 14:21:33 +0800 Ready: True Restart Count: 0 Requests: cpu: 10m Environment: POD_NAME: activator-55c4768746-5bpxk (v1:metadata.name) POD_NAMESPACE: knative-serving (v1:metadata.namespace) INSTANCE_IP: (v1:status.podIP) ISTIO_META_POD_NAME: activator-55c4768746-5bpxk (v1:metadata.name) ISTIO_META_INTERCEPTION_MODE: REDIRECT ISTIO_METAJSON_ANNOTATIONS: {"sidecar.istio.io/inject":"true"}

  ISTIO_METAJSON_LABELS:         {"app":"activator","pod-template-hash":"55c4768746","role":"activator","serving.knative.dev/release":"devel"}

Mounts:
  /etc/certs/ from istio-certs (ro)
  /etc/istio/proxy from istio-envoy (rw)

Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: config-logging: Type: ConfigMap (a volume populated by a ConfigMap) Name: config-logging Optional: false config-observability: Type: ConfigMap (a volume populated by a ConfigMap) Name: config-observability Optional: false controller-token-rkmfr: Type: Secret (a volume populated by a Secret) SecretName: controller-token-rkmfr Optional: false istio-envoy: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory istio-certs: Type: Secret (a volume populated by a Secret) SecretName: istio.controller Optional: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message

Warning Unhealthy 7m54s (x99 over 125m) kubelet, aks-agentpool-40400703-3 Readiness probe failed: HTTP probe failed with statuscode: 503 Warning BackOff 2m44s (x489 over 123m) kubelet, aks-agentpool-40400703-3 Back-off restarting failed container

`

Question: what does the code 503 mean?

xpepermint commented 5 years ago

+1 on knative 0.6.0, istio 1.1.8 (there are changes from v1.0.0, but also tried with 1.0.8), Flannel network fabric, https://github.com/knative/serving/issues/3903

xpepermint commented 5 years ago

More info: Happens on Ubuntu 18.04 but not on 19.04.

magicLian commented 5 years ago

For me , I install the Istio in PMTLS to solve it .

https://istio.io/docs/setup/kubernetes/install/kubernetes/

yuxiaoba commented 5 years ago

@magicLian In your url for installing istio, there only have permissive mutual TLS install and strict mutual TLS install。 How can I install the Istio without MTLS ?

magicLian commented 5 years ago

@yuxiaoba Sorry to trouble u , i install Istio in pmtls mode.

yuxiaoba commented 5 years ago

@magicLian Can you give me a guide for how to install Istio in pmtls mode? I am a fresh man for this , thank you.

magicLian commented 5 years ago

@yuxiaoba 就是用官方的例子呀， kubectl apply -f install/kubernetes/istio-demo.yaml 你进我之前发的安装istio的页面，你就能看到。

yuxiaoba commented 5 years ago

@magicLian 噢噢，我也是用那个安装的，但是也还是持续CrashLoopBackOff。你的Kubernetes版本是多少啊？

magicLian commented 5 years ago

@yuxiaoba 1.12.6

mattmoor commented 5 years ago

This looks like: https://github.com/knative/serving/issues/4407?

mattmoor commented 5 years ago

@yuxiaoba ^^

yuxiaoba commented 5 years ago

@mattmoor Yes， they looks like the same problem. I also try to install istio:

kubectl apply -f install/kubernetes/istio-demo.yaml

But it dose not help. The activator keeps carshloopbackoff.

mattmoor commented 5 years ago

Let's discuss there.