k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
28.1k stars 2.35k forks source link

Resolve errors with installing Knative #286

Closed erikwilson closed 5 years ago

erikwilson commented 5 years ago

Describe the bug Errors running Knative

To Reproduce Steps to reproduce the behavior: https://www.knative.dev/docs/install/knative-with-iks/#installing-istio

Screenshots Error Step 1:

unable to recognize "https://github.com/knative/serving/releases/download/v0.4.0/istio.yaml": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta1"

And when installing Knative Step 2:

unable to recognize "https://github.com/knative/eventing/releases/download/v0.4.0/release.yaml": no matches for kind "ClusterChannelProvisioner" in version "eventing.knative.dev/v1alpha1"
root@erik-k3s-dev:~# kubectl get pods --all-namespaces
NAMESPACE            NAME                                            READY   STATUS             RESTARTS   AGE
istio-system         cluster-local-gateway-547467ccf6-7xbxw          1/1     Running            0          7m58s
istio-system         istio-citadel-7d64db8bcf-vvvc5                  1/1     Running            0          7m59s
istio-system         istio-cleanup-secrets-hfwww                     0/1     Completed          0          8m
istio-system         istio-egressgateway-6ddf4c8bd6-k8l94            1/1     Running            0          8m
istio-system         istio-galley-7dd996474-kdxj2                    1/1     Running            0          8m
istio-system         istio-ingressgateway-84b89d647f-th8bk           1/1     Running            0          8m
istio-system         istio-pilot-86bb4fcbbd-p7s6c                    2/2     Running            0          7m59s
istio-system         istio-policy-5c4d9ff96b-xzkbl                   2/2     Running            0          7m59s
istio-system         istio-sidecar-injector-6977b5cf5b-g85j4         1/1     Running            0          7m59s
istio-system         istio-statsd-prom-bridge-b44b96d7b-fzk8l        1/1     Running            0          8m
istio-system         istio-telemetry-7676df547f-5xmpd                2/2     Running            0          7m59s
istio-system         zipkin-5495cf4cf-jxf2k                          1/1     Running            0          6m54s
knative-build        build-controller-7b8987d675-mvtkv               1/1     Running            0          6m56s
knative-build        build-webhook-74795c8696-564zr                  1/1     Running            0          6m56s
knative-eventing     eventing-controller-864657d8d4-wqnch            1/1     Running            0          6m56s
knative-eventing     in-memory-channel-controller-f794cc9d8-g2qv4    1/1     Running            0          6m55s
knative-eventing     in-memory-channel-dispatcher-8595c7f8d7-wtvjs   1/2     CrashLoopBackOff   6          6m55s
knative-eventing     webhook-5d76776d55-z9wpm                        1/1     Running            0          6m56s
knative-monitoring   elasticsearch-logging-0                         1/1     Running            0          6m55s
knative-monitoring   elasticsearch-logging-1                         1/1     Running            0          5m50s
knative-monitoring   grafana-568674f4f9-p257v                        1/1     Running            0          6m54s
knative-monitoring   kibana-logging-7698db4f94-frdmb                 1/1     Running            0          6m55s
knative-monitoring   kube-state-metrics-58b9779756-bbvsd             4/4     Running            0          5m46s
knative-monitoring   node-exporter-ftfzq                             2/2     Running            0          6m55s
knative-monitoring   prometheus-system-0                             1/1     Running            0          6m54s
knative-monitoring   prometheus-system-1                             1/1     Running            0          6m54s
knative-serving      activator-7c8b59d78-gkhnw                       1/2     CrashLoopBackOff   6          6m57s
knative-serving      autoscaler-666c9bfcc6-5q6z7                     1/2     CrashLoopBackOff   6          6m57s
knative-serving      controller-799cd5c6dc-sr9md                     1/1     Running            0          6m57s
knative-serving      webhook-5b66fdf6b9-ssg84                        1/1     Running            0          6m57s
knative-sources      controller-manager-0                            1/1     Running            0          6m55s
kube-system          coredns-7748f7f6df-m7jm9                        1/1     Running            0          8m42s
kube-system          helm-install-traefik-d8j6d                      0/1     Completed          0          8m42s
kube-system          svclb-traefik-8f7c97697-s5wqd                   2/2     Running            1          8m31s
kube-system          traefik-6876857645-dl88l                        1/1     Running            0          8m31s
root@erik-k3s-dev:~# kubectl logs -n knative-eventing in-memory-channel-dispatcher-8595c7f8d7-wtvjs dispatcher
{"level":"fatal","ts":1552001051.4944558,"caller":"fanoutsidecar/main.go:91","msg":"Unable to create configMap noticer.","error":"Get https://10.43.0.1:443/api?timeout=32s: EOF","stacktrace":"main.main\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:91\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"}
root@erik-k3s-dev:~# kubectl logs -n knative-serving activator-7c8b59d78-gkhnw activator
{"level":"info","ts":"2019-03-07T23:25:49.551Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}"}
{"level":"info","ts":"2019-03-07T23:25:49.551Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-03-07T23:25:49.551Z","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.4\" is not a valid GitHub commit ID"}
{"level":"info","ts":"2019-03-07T23:25:49.551Z","logger":"activator","caller":"activator/main.go:131","msg":"Starting the knative activator","knative.dev/controller":"activator"}
{"level":"fatal","ts":"2019-03-07T23:25:51.644Z","logger":"activator","caller":"activator/main.go:147","msg":"Version check failed: Get https://10.43.0.1:443/version?timeout=32s: EOF","knative.dev/controller":"activator","stacktrace":"main.main\n\t/go/src/github.com/knative/serving/cmd/activator/main.go:147\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"}
root@erik-k3s-dev:~# kubectl logs -n knative-serving autoscaler-666c9bfcc6-5q6z7 autoscaler
{"level":"info","ts":"2019-03-07T23:25:24.499Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}"}
{"level":"info","ts":"2019-03-07T23:25:24.499Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-03-07T23:25:24.499Z","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.4\" is not a valid GitHub commit ID"}
{"level":"fatal","ts":"2019-03-07T23:25:33.647Z","logger":"autoscaler","caller":"autoscaler/main.go:95","msg":"Version check failed: an error on the server (\"\") has prevented the request from succeeding","stacktrace":"main.main\n\t/go/src/github.com/knative/serving/cmd/autoscaler/main.go:95\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"}
root@erik-k3s-dev:~# head -3 /proc/meminfo
MemTotal:       32939856 kB
MemFree:        16705668 kB
MemAvailable:   23599904 kB

Additional context rancher/k3s/issues/90 rancher/k3s/issues/285

galal-hussein commented 5 years ago

I don't see any errors any more with the latest k3s, for the release error it seems to be a race condition happens because crds for knative wasn't initialized, here is the original issue https://github.com/knative/eventing/issues/680

Running the command again works just fine.

erikwilson commented 5 years ago

I'm still seeing some issues with certain pods, running with knative 0.5.0 and k3s v0.5.0-rc1:

[ 2019-04-26 16:37:28 ]
root🐮k3s:~$ kubectl get pods -A
NAMESPACE            NAME                                            READY   STATUS             RESTARTS   AGE
istio-system         cluster-local-gateway-5bf5488bb-f5x42           1/1     Running            0          6m59s
istio-system         istio-citadel-78fd647cf-cb2b4                   1/1     Running            0          7m
istio-system         istio-cleanup-secrets-gfqtd                     0/1     Completed          0          6m59s
istio-system         istio-egressgateway-5547468b8d-6p4mt            1/1     Running            0          7m
istio-system         istio-galley-64ddbbff8b-cjjbk                   1/1     Running            0          7m
istio-system         istio-ingressgateway-686d54b4bf-6qsdl           1/1     Running            0          7m
istio-system         istio-pilot-7bd454bf5c-g2cvr                    2/2     Running            0          7m
istio-system         istio-pilot-7bd454bf5c-lgfg8                    2/2     Running            0          6m45s
istio-system         istio-pilot-7bd454bf5c-vztzx                    2/2     Running            0          6m45s
istio-system         istio-policy-64b5ff6bd8-2dtsn                   2/2     Running            0          7m
istio-system         istio-security-post-install-2t84k               0/1     Completed          0          6m59s
istio-system         istio-sidecar-injector-6956c47586-t2phg         1/1     Running            0          7m
istio-system         istio-telemetry-55b97c59c4-w25kj                2/2     Running            0          7m
istio-system         svclb-istio-ingressgateway-68sqr                0/8     Pending            0          7m1s
istio-system         zipkin-57869dbb48-k7ld2                         1/1     Running            0          6m16s
knative-build        build-controller-77cbbc9888-dw2j9               1/1     Running            0          5m37s
knative-build        build-webhook-64f9bbc7c9-qtzqx                  1/1     Running            0          5m28s
knative-eventing     eventing-controller-f747584b7-zwb9c             1/1     Running            0          5m28s
knative-eventing     in-memory-channel-controller-5f587cc68c-x4q5g   1/1     Running            0          5m37s
knative-eventing     in-memory-channel-dispatcher-9f7d664d4-8k9k7    1/2     CrashLoopBackOff   5          5m37s
knative-eventing     webhook-5fd955f896-msszp                        1/1     Running            0          5m37s
knative-monitoring   elasticsearch-logging-0                         0/1     CrashLoopBackOff   3          5m37s
knative-monitoring   elasticsearch-logging-1                         0/1     CrashLoopBackOff   3          3m40s
knative-monitoring   grafana-69f88c5544-ggrdn                        1/1     Running            0          5m37s
knative-monitoring   kibana-logging-7cb6b64bff-tvsw7                 1/1     Running            0          5m37s
knative-monitoring   kube-state-metrics-75fdd67ffc-7jpph             4/4     Running            0          4m21s
knative-monitoring   node-exporter-zqn67                             2/2     Running            0          5m20s
knative-monitoring   prometheus-system-0                             1/1     Running            0          5m37s
knative-monitoring   prometheus-system-1                             1/1     Running            0          5m37s
knative-serving      activator-654b696b84-zmkkq                      1/2     CrashLoopBackOff   5          5m28s
knative-serving      autoscaler-585b674986-lrvsk                     2/2     Running            0          5m28s
knative-serving      controller-54b89c944c-qh6fc                     1/1     Running            0          5m37s
knative-serving      webhook-5cc857b9b8-vps79                        1/1     Running            0          5m28s
knative-sources      controller-manager-0                            1/1     Running            0          5m34s
kube-system          coredns-695688789-xbkb5                         1/1     Running            0          7m33s
kube-system          helm-install-traefik-5mv4v                      0/1     Completed          0          7m33s
kube-system          svclb-traefik-4b7g9                             2/2     Running            0          7m14s
kube-system          traefik-55bd9646fc-bxgqj                        1/1     Running            0          7m14s
[ 2019-04-26 16:37:43 ]
root🐮k3s:~$ kubectl logs -n knative-eventing service/in-memory-channel-dispatcher dispatcher
{"level":"error","ts":"2019-04-26T23:36:38.015Z","caller":"fanoutsidecar/main.go:131","msg":"Error starting manager.","error":"Get https://10.43.0.1:443/api?timeout=32s: EOF","stacktrace":"main.setupConfigMapNoticer\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:131\nmain.main\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"fatal","ts":"2019-04-26T23:36:38.015Z","caller":"fanoutsidecar/main.go:94","msg":"Unable to create configMap noticer.","error":"Get https://10.43.0.1:443/api?timeout=32s: EOF","stacktrace":"main.main\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:94\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
[ 2019-04-26 16:37:46 ]
root🐮k3s:~$ kubectl logs -n knative-serving deployment.apps/activator activator | head -5
{"level":"info","ts":"2019-04-26T23:36:24.823Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}"}
{"level":"info","ts":"2019-04-26T23:36:24.823Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-04-26T23:36:24.823Z","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.5\" is not a valid GitHub commit ID"}
{"level":"info","ts":"2019-04-26T23:36:24.823Z","logger":"activator","caller":"activator/main.go:134","msg":"Starting the knative activator","knative.dev/controller":"activator"}
{"level":"error","ts":"2019-04-26T23:36:26.827Z","logger":"activator","caller":"activator/main.go:152","msg":"Failed to get k8s version","knative.dev/controller":"activator","error":"Get https://10.43.0.1:443/version?timeout=32s: EOF","stacktrace":"main.main.func1\n\t/go/src/github.com/knative/serving/cmd/activator/main.go:152\ngithub.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait.pollImmediateInternal\n\t/go/src/github.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:245\ngithub.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/go/src/github.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:241\nmain.main\n\t/go/src/github.com/knative/serving/cmd/activator/main.go:150\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
erikwilson commented 5 years ago

Should be fixed with #359 where the advertise ip is set to the node ip.

galal-hussein commented 5 years ago

k3s version: v0.7.0-rc6

I was able to verify that knative and istio installation worked successfully:

# kubectl get pods --namespace knative-serving
NAME                                READY   STATUS    RESTARTS   AGE
activator-5ccf59c56-z85fc           1/1     Running   0          96s
autoscaler-7f55589d4b-77pb9         1/1     Running   0          96s
controller-77cd5ffdc4-qgdz7         1/1     Running   0          96s
networking-istio-5476685c6f-4xtcf   1/1     Running   0          96s
webhook-5f668cf9f8-zpwng            1/1     Running   0          96s

# kubectl get pods --namespace knative-build
NAME                               READY   STATUS    RESTARTS   AGE
build-controller-f969bf74d-7rgg6   1/1     Running   0          96s
build-webhook-6f58484688-l4994     1/1     Running   0          96s

# kubectl get pods --namespace knative-eventing
NAME                                            READY   STATUS    RESTARTS   AGE
eventing-controller-569448686c-t4s9f            1/1     Running   0          95s
eventing-webhook-6d6795d497-2rv87               1/1     Running   0          95s
imc-controller-77f4776c99-mn4z8                 1/1     Running   0          95s
imc-dispatcher-7b69ccd5d8-mqgkx                 1/1     Running   0          95s
in-memory-channel-controller-85dc669f69-vjqrm   1/1     Running   0          95s
in-memory-channel-dispatcher-7f896c7d85-4tmsh   1/1     Running   0          95s
sources-controller-7b7579bc94-9kgr7             1/1     Running   0          95s

# kubectl get pods --namespace knative-monitoring
NAME                                  READY   STATUS    RESTARTS   AGE
elasticsearch-logging-0               1/1     Running   0          95s
elasticsearch-logging-1               1/1     Running   0          22s
grafana-5b8fcf6dbd-79982              1/1     Running   0          93s
kibana-logging-7cb6b64bff-26brf       1/1     Running   0          94s
kube-state-metrics-75fdd67ffc-zwmb8   4/4     Running   0          45s
node-exporter-wftcq                   2/2     Running   0          94s
prometheus-system-0                   1/1     Running   0          93s
prometheus-system-1                   1/1     Running   0          93s
beriberikix commented 5 years ago

Can you share more details on the steps you took to install Istio (and Knative?) I've been running into issues where the Istio pods are stuck in pending. I've started a thread in the Istio Slack.

Note: I'm running on a Raspberry Pi (ARM64) cluster set up with K3sup.

erikwilson commented 5 years ago

It looks like istio may be broken with k8s 1.16 from the knative instructions, where some of the beta APIs have been removed.

beriberikix commented 5 years ago

Oh interesting, you mean these instructions? Any proposed changes?

FWIW, it seems like the latest k3sup doesn't use 1.16.

jonathanberi:istio-1.3.3 🚀 $ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:51:13Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4-k3s.1", GitCommit:"a7531b1ab3fd5ff987b074472ddfb84a2f5326bc", GitTreeState:"clean", BuildDate:"2019-09-19T22:36Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/arm"}
erikwilson commented 5 years ago

It looks like istio has node selectors for amd64, ppc64le, or s390x architectures (quick look at docker hub seems they only publish for amd64), so makes sense that arm64 or arm would remain in pending. :(

xgenvn commented 5 years ago

I believe many of us willing to try out Knative since it's a very active project. I'll be great if we have a documentation section or wiki on this as a tutorial.

I'm also stucking installing knative using glooctl. I remove the traefik from system namespace, and the installation of Knative+Gloo goes fine, but there is one blocking issue with gloo proxy that I can't connect to. There is some missing points but I don't know where to start to research.

I will try with Istio to see everything going fine.

amio commented 4 years ago

I'v encountered same problem with knative 0.13.0:

$ kubectl apply --filename https://github.com/knative/serving/releases/download/v0.13.0/serving-istio.yaml                                                             15:03:18
clusterrole.rbac.authorization.k8s.io/knative-serving-istio unchanged
configmap/config-istio unchanged
deployment.apps/networking-istio unchanged
unable to recognize "https://github.com/knative/serving/releases/download/v0.13.0/serving-istio.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
unable to recognize "https://github.com/knative/serving/releases/download/v0.13.0/serving-istio.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4+k3s1", GitCommit:"3eee8ac3a1cf0a216c8a660571329d4bda3bdf77", GitTreeState:"clean", BuildDate:"2020-03-25T16:13:25Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4+k3s1", GitCommit:"3eee8ac3a1cf0a216c8a660571329d4bda3bdf77", GitTreeState:"clean", BuildDate:"2020-03-25T16:13:25Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

$ docker -v
Docker version 17.06.2-ce, build 0e7f239

$ k3s --version
k3s version v1.17.4+k3s1 (3eee8ac3)

It's a two nodes cluster:

$ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
n227-160-185   Ready    master   28h   v1.17.4+k3s1
n227-160-033   Ready    master   26h   v1.17.4+k3s1
brandond commented 4 years ago

@amio sounds like your Istio CRDs are missing or incompatible. Which version of Istio did you install?

Possibly related: https://github.com/knative/serving/issues/2195

amio commented 4 years ago

@brandond I was using this to install it:

$ kubectl apply --filename https://github.com/knative/serving/releases/download/v0.13.0/serving-crds.yaml                                                     14:58:14
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
brandond commented 4 years ago

@amio that doesn't include any of the istio CRDs, just the knative CRDs. Did you maybe need to upgrade istio before upgrading knative? It looks like it wants 1.4.6: https://knative.dev/development/install/installing-istio/

amio commented 4 years ago

@brondond Thanks for pointing out! I did miss the step of installing Istio 😆