Closed erikwilson closed 5 years ago
I don't see any errors any more with the latest k3s, for the release error it seems to be a race condition happens because crds for knative wasn't initialized, here is the original issue https://github.com/knative/eventing/issues/680
Running the command again works just fine.
I'm still seeing some issues with certain pods, running with knative 0.5.0 and k3s v0.5.0-rc1:
[ 2019-04-26 16:37:28 ]
root🐮k3s:~$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
istio-system cluster-local-gateway-5bf5488bb-f5x42 1/1 Running 0 6m59s
istio-system istio-citadel-78fd647cf-cb2b4 1/1 Running 0 7m
istio-system istio-cleanup-secrets-gfqtd 0/1 Completed 0 6m59s
istio-system istio-egressgateway-5547468b8d-6p4mt 1/1 Running 0 7m
istio-system istio-galley-64ddbbff8b-cjjbk 1/1 Running 0 7m
istio-system istio-ingressgateway-686d54b4bf-6qsdl 1/1 Running 0 7m
istio-system istio-pilot-7bd454bf5c-g2cvr 2/2 Running 0 7m
istio-system istio-pilot-7bd454bf5c-lgfg8 2/2 Running 0 6m45s
istio-system istio-pilot-7bd454bf5c-vztzx 2/2 Running 0 6m45s
istio-system istio-policy-64b5ff6bd8-2dtsn 2/2 Running 0 7m
istio-system istio-security-post-install-2t84k 0/1 Completed 0 6m59s
istio-system istio-sidecar-injector-6956c47586-t2phg 1/1 Running 0 7m
istio-system istio-telemetry-55b97c59c4-w25kj 2/2 Running 0 7m
istio-system svclb-istio-ingressgateway-68sqr 0/8 Pending 0 7m1s
istio-system zipkin-57869dbb48-k7ld2 1/1 Running 0 6m16s
knative-build build-controller-77cbbc9888-dw2j9 1/1 Running 0 5m37s
knative-build build-webhook-64f9bbc7c9-qtzqx 1/1 Running 0 5m28s
knative-eventing eventing-controller-f747584b7-zwb9c 1/1 Running 0 5m28s
knative-eventing in-memory-channel-controller-5f587cc68c-x4q5g 1/1 Running 0 5m37s
knative-eventing in-memory-channel-dispatcher-9f7d664d4-8k9k7 1/2 CrashLoopBackOff 5 5m37s
knative-eventing webhook-5fd955f896-msszp 1/1 Running 0 5m37s
knative-monitoring elasticsearch-logging-0 0/1 CrashLoopBackOff 3 5m37s
knative-monitoring elasticsearch-logging-1 0/1 CrashLoopBackOff 3 3m40s
knative-monitoring grafana-69f88c5544-ggrdn 1/1 Running 0 5m37s
knative-monitoring kibana-logging-7cb6b64bff-tvsw7 1/1 Running 0 5m37s
knative-monitoring kube-state-metrics-75fdd67ffc-7jpph 4/4 Running 0 4m21s
knative-monitoring node-exporter-zqn67 2/2 Running 0 5m20s
knative-monitoring prometheus-system-0 1/1 Running 0 5m37s
knative-monitoring prometheus-system-1 1/1 Running 0 5m37s
knative-serving activator-654b696b84-zmkkq 1/2 CrashLoopBackOff 5 5m28s
knative-serving autoscaler-585b674986-lrvsk 2/2 Running 0 5m28s
knative-serving controller-54b89c944c-qh6fc 1/1 Running 0 5m37s
knative-serving webhook-5cc857b9b8-vps79 1/1 Running 0 5m28s
knative-sources controller-manager-0 1/1 Running 0 5m34s
kube-system coredns-695688789-xbkb5 1/1 Running 0 7m33s
kube-system helm-install-traefik-5mv4v 0/1 Completed 0 7m33s
kube-system svclb-traefik-4b7g9 2/2 Running 0 7m14s
kube-system traefik-55bd9646fc-bxgqj 1/1 Running 0 7m14s
[ 2019-04-26 16:37:43 ]
root🐮k3s:~$ kubectl logs -n knative-eventing service/in-memory-channel-dispatcher dispatcher
{"level":"error","ts":"2019-04-26T23:36:38.015Z","caller":"fanoutsidecar/main.go:131","msg":"Error starting manager.","error":"Get https://10.43.0.1:443/api?timeout=32s: EOF","stacktrace":"main.setupConfigMapNoticer\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:131\nmain.main\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"fatal","ts":"2019-04-26T23:36:38.015Z","caller":"fanoutsidecar/main.go:94","msg":"Unable to create configMap noticer.","error":"Get https://10.43.0.1:443/api?timeout=32s: EOF","stacktrace":"main.main\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:94\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
[ 2019-04-26 16:37:46 ]
root🐮k3s:~$ kubectl logs -n knative-serving deployment.apps/activator activator | head -5
{"level":"info","ts":"2019-04-26T23:36:24.823Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n \"level\": \"info\",\n \"development\": false,\n \"outputPaths\": [\"stdout\"],\n \"errorOutputPaths\": [\"stderr\"],\n \"encoding\": \"json\",\n \"encoderConfig\": {\n \"timeKey\": \"ts\",\n \"levelKey\": \"level\",\n \"nameKey\": \"logger\",\n \"callerKey\": \"caller\",\n \"messageKey\": \"msg\",\n \"stacktraceKey\": \"stacktrace\",\n \"lineEnding\": \"\",\n \"levelEncoder\": \"\",\n \"timeEncoder\": \"iso8601\",\n \"durationEncoder\": \"\",\n \"callerEncoder\": \"\"\n }\n}"}
{"level":"info","ts":"2019-04-26T23:36:24.823Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-04-26T23:36:24.823Z","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.5\" is not a valid GitHub commit ID"}
{"level":"info","ts":"2019-04-26T23:36:24.823Z","logger":"activator","caller":"activator/main.go:134","msg":"Starting the knative activator","knative.dev/controller":"activator"}
{"level":"error","ts":"2019-04-26T23:36:26.827Z","logger":"activator","caller":"activator/main.go:152","msg":"Failed to get k8s version","knative.dev/controller":"activator","error":"Get https://10.43.0.1:443/version?timeout=32s: EOF","stacktrace":"main.main.func1\n\t/go/src/github.com/knative/serving/cmd/activator/main.go:152\ngithub.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait.pollImmediateInternal\n\t/go/src/github.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:245\ngithub.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/go/src/github.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:241\nmain.main\n\t/go/src/github.com/knative/serving/cmd/activator/main.go:150\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
Should be fixed with #359 where the advertise ip is set to the node ip.
k3s version: v0.7.0-rc6
I was able to verify that knative and istio installation worked successfully:
# kubectl get pods --namespace knative-serving
NAME READY STATUS RESTARTS AGE
activator-5ccf59c56-z85fc 1/1 Running 0 96s
autoscaler-7f55589d4b-77pb9 1/1 Running 0 96s
controller-77cd5ffdc4-qgdz7 1/1 Running 0 96s
networking-istio-5476685c6f-4xtcf 1/1 Running 0 96s
webhook-5f668cf9f8-zpwng 1/1 Running 0 96s
# kubectl get pods --namespace knative-build
NAME READY STATUS RESTARTS AGE
build-controller-f969bf74d-7rgg6 1/1 Running 0 96s
build-webhook-6f58484688-l4994 1/1 Running 0 96s
# kubectl get pods --namespace knative-eventing
NAME READY STATUS RESTARTS AGE
eventing-controller-569448686c-t4s9f 1/1 Running 0 95s
eventing-webhook-6d6795d497-2rv87 1/1 Running 0 95s
imc-controller-77f4776c99-mn4z8 1/1 Running 0 95s
imc-dispatcher-7b69ccd5d8-mqgkx 1/1 Running 0 95s
in-memory-channel-controller-85dc669f69-vjqrm 1/1 Running 0 95s
in-memory-channel-dispatcher-7f896c7d85-4tmsh 1/1 Running 0 95s
sources-controller-7b7579bc94-9kgr7 1/1 Running 0 95s
# kubectl get pods --namespace knative-monitoring
NAME READY STATUS RESTARTS AGE
elasticsearch-logging-0 1/1 Running 0 95s
elasticsearch-logging-1 1/1 Running 0 22s
grafana-5b8fcf6dbd-79982 1/1 Running 0 93s
kibana-logging-7cb6b64bff-26brf 1/1 Running 0 94s
kube-state-metrics-75fdd67ffc-zwmb8 4/4 Running 0 45s
node-exporter-wftcq 2/2 Running 0 94s
prometheus-system-0 1/1 Running 0 93s
prometheus-system-1 1/1 Running 0 93s
Can you share more details on the steps you took to install Istio (and Knative?) I've been running into issues where the Istio pods are stuck in pending. I've started a thread in the Istio Slack.
Note: I'm running on a Raspberry Pi (ARM64) cluster set up with K3sup.
It looks like istio may be broken with k8s 1.16 from the knative instructions, where some of the beta APIs have been removed.
Oh interesting, you mean these instructions? Any proposed changes?
FWIW, it seems like the latest k3sup doesn't use 1.16.
jonathanberi:istio-1.3.3 🚀 $ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:51:13Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4-k3s.1", GitCommit:"a7531b1ab3fd5ff987b074472ddfb84a2f5326bc", GitTreeState:"clean", BuildDate:"2019-09-19T22:36Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/arm"}
It looks like istio has node selectors for amd64, ppc64le, or s390x architectures (quick look at docker hub seems they only publish for amd64), so makes sense that arm64 or arm would remain in pending. :(
I believe many of us willing to try out Knative since it's a very active project. I'll be great if we have a documentation section or wiki on this as a tutorial.
I'm also stucking installing knative using glooctl. I remove the traefik from system namespace, and the installation of Knative+Gloo goes fine, but there is one blocking issue with gloo proxy that I can't connect to. There is some missing points but I don't know where to start to research.
I will try with Istio to see everything going fine.
I'v encountered same problem with knative 0.13.0:
$ kubectl apply --filename https://github.com/knative/serving/releases/download/v0.13.0/serving-istio.yaml 15:03:18
clusterrole.rbac.authorization.k8s.io/knative-serving-istio unchanged
configmap/config-istio unchanged
deployment.apps/networking-istio unchanged
unable to recognize "https://github.com/knative/serving/releases/download/v0.13.0/serving-istio.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
unable to recognize "https://github.com/knative/serving/releases/download/v0.13.0/serving-istio.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4+k3s1", GitCommit:"3eee8ac3a1cf0a216c8a660571329d4bda3bdf77", GitTreeState:"clean", BuildDate:"2020-03-25T16:13:25Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4+k3s1", GitCommit:"3eee8ac3a1cf0a216c8a660571329d4bda3bdf77", GitTreeState:"clean", BuildDate:"2020-03-25T16:13:25Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
$ docker -v
Docker version 17.06.2-ce, build 0e7f239
$ k3s --version
k3s version v1.17.4+k3s1 (3eee8ac3)
It's a two nodes cluster:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
n227-160-185 Ready master 28h v1.17.4+k3s1
n227-160-033 Ready master 26h v1.17.4+k3s1
@amio sounds like your Istio CRDs are missing or incompatible. Which version of Istio did you install?
Possibly related: https://github.com/knative/serving/issues/2195
@brandond I was using this to install it:
$ kubectl apply --filename https://github.com/knative/serving/releases/download/v0.13.0/serving-crds.yaml 14:58:14
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
@amio that doesn't include any of the istio CRDs, just the knative CRDs. Did you maybe need to upgrade istio before upgrading knative? It looks like it wants 1.4.6: https://knative.dev/development/install/installing-istio/
@brondond Thanks for pointing out! I did miss the step of installing Istio 😆
Describe the bug Errors running Knative
To Reproduce Steps to reproduce the behavior: https://www.knative.dev/docs/install/knative-with-iks/#installing-istio
Screenshots Error Step 1:
And when installing Knative Step 2:
Additional context rancher/k3s/issues/90 rancher/k3s/issues/285