search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.49k stars 773 forks source link

knative not working on microk8s 1.18 (Ubuntu 20.04) #1517

Closed ayqazi closed 3 years ago

ayqazi commented 4 years ago

I am following the instructions to install microk8s and then knative, and then installing a sample knative app. The deployment seems to occur fine but then when I inspect the running services, they cannot seem to access the internet to download the image I pushed to Docker Hub. It is definitely a public image.

I am running Ubuntu 20.04 (Ubuntu MATE flavour). I install microk8s using snap, and it installs 1.18.6. microk8s installs knative 0.17.

I followed the steps to run knative on microk8s here: https://ubuntu.com/blog/getting-started-with-knative-1

It was quite out of date but the steps seemed to work. Here is what I did to install it:

sudo snap install microk8s --classic
=> microk8s v1.18.6 from Canonical✓ installed

microk8s status --wait-ready
sudo snap alias microk8s.kubectl kubectl

sudo ufw allow in on cni0 && sudo ufw allow out on cni0 && sudo ufw default allow routed

sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube

# log out and in again
groups # shows I'm in microk8s group

microk8s enable knative
microk8s status --wait-ready # istio, dns, knative enabled
kubectl get all --all-namespaces # all working
kubectl get pods --all-namespaces | grep knative # Quick check

Now I want to deploy the sample app here: https://github.com/knative/docs/blob/release-0.17/docs/serving/samples/hello-world/helloworld-ruby/README.md

<check out project>
cd <project dir>/docs/serving/samples/hello-world/helloworld-ruby
sed -i -e 's/{username}/ayqazi/' service.yaml
docker build -t ayqazi/helloworld-ruby .
docker push ayqazi/helloworld-ruby

kubectl apply --filename service.yaml

It seems to deploy successfully.

However now comes the problem:

$ kubectl get ksvc helloworld-ruby
=> NAME              URL                                          LATESTCREATED           LATESTREADY   READY     REASON
=> helloworld-ruby   http://helloworld-ruby.default.example.com   helloworld-ruby-24np4                 Unknown   RevisionMissing

There's no xip.io URL like I was expecting, plus that error.

I dig more:

$ kubectl get all --namespace default
NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                             AGE
service/helloworld-ruby-24np4           ClusterIP   10.152.183.234   <none>        80/TCP                              2m57s
service/helloworld-ruby-24np4-private   ClusterIP   10.152.183.188   <none>        80/TCP,9090/TCP,9091/TCP,8022/TCP   2m57s
service/kubernetes                      ClusterIP   10.152.183.1     <none>        443/TCP                             16m

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/helloworld-ruby-24np4-deployment   0/0     0            0           2m57s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/helloworld-ruby-24np4-deployment-55b798d75b   0         0         0       2m57s

NAME                                          URL                                          LATESTCREATED           LATESTREADY   READY     REASON
service.serving.knative.dev/helloworld-ruby   http://helloworld-ruby.default.example.com   helloworld-ruby-24np4                 Unknown   RevisionMissing

NAME                                                 CONFIG NAME       K8S SERVICE NAME        GENERATION   READY     REASON
revision.serving.knative.dev/helloworld-ruby-24np4   helloworld-ruby   helloworld-ruby-24np4   1            Unknown   Deploying

NAME                                        URL                                          READY     REASON
route.serving.knative.dev/helloworld-ruby   http://helloworld-ruby.default.example.com   Unknown   RevisionMissing

NAME                                                LATESTCREATED           LATESTREADY   READY     REASON
configuration.serving.knative.dev/helloworld-ruby   helloworld-ruby-24np4                 Unknown

I found this in the logs but do not know enough to understand what it means:

$ sudo tail /var/log/pods/knative-serving_controller-7ffb84fd9c-d6gmw_bbdc6cc3-ba19-4c3b-8154-0580a179d3e1/controller/1.log 
2020-08-26T16:18:38.384959634+01:00 stderr F 2020/08/26 15:18:38 Registering 4 clients
2020-08-26T16:18:38.384986028+01:00 stderr F 2020/08/26 15:18:38 Registering 3 informer factories
2020-08-26T16:18:38.38498979+01:00 stderr F 2020/08/26 15:18:38 Registering 13 informers
2020-08-26T16:18:38.384991945+01:00 stderr F 2020/08/26 15:18:38 Registering 7 controllers
2020-08-26T16:18:38.386013104+01:00 stderr F 2020/08/26 15:18:38 Error reading/parsing logging configuration: Get https://10.152.183.1:443/api/v1/namespaces/knative-serving/configmaps/config-logging: dial tcp 10.152.183.1:443: connect: connection refused

microk8s inspect:

Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-flanneld is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/1551/inspection-report-20200826_163103.tar.gz

Said tarball attached.

inspection-report-20200826_163103.tar.gz

I'd be grateful for your aid in the matter.

Thanks

ktsakalozos commented 4 years ago

Hi @ayqazi

This issue looks similar to https://github.com/ubuntu/microk8s/issues/784

Would you care reinstalling microk8s from the latest/edge/runc-nnp channel as described in https://github.com/ubuntu/microk8s/issues/784#issuecomment-680781229

It would be great to get some feedback on the fix so we can proceed with merging and releasing it. Thank you.

ayqazi commented 4 years ago

Hi @ktsakalozos,

The version you asked me to use gave a new error:

$ kubectl get all --namespace default
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/helloworld-ruby-f8cdg-deployment-5fdf695577-m86cp   2/2     Running   0          51s

NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP                           PORT(S)                             AGE
service/kubernetes                      ClusterIP      10.152.183.1     <none>                                443/TCP                             6m19s
service/helloworld-ruby-f8cdg-private   ClusterIP      10.152.183.73    <none>                                80/TCP,9090/TCP,9091/TCP,8022/TCP   51s
service/helloworld-ruby-f8cdg           ClusterIP      10.152.183.212   <none>                                80/TCP                              51s
service/helloworld-ruby                 ExternalName   <none>           helloworld-ruby.default.example.com   <none>                              36s

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/helloworld-ruby-f8cdg-deployment   1/1     1            1           51s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/helloworld-ruby-f8cdg-deployment-5fdf695577   1         1         1       51s

NAME                                                 CONFIG NAME       K8S SERVICE NAME        GENERATION   READY   REASON
revision.serving.knative.dev/helloworld-ruby-f8cdg   helloworld-ruby   helloworld-ruby-f8cdg   1            True    

NAME                                          URL                                          LATESTCREATED           LATESTREADY             READY     REASON
service.serving.knative.dev/helloworld-ruby   http://helloworld-ruby.default.example.com   helloworld-ruby-f8cdg   helloworld-ruby-f8cdg   Unknown   IngressNotConfigured

NAME                                        URL                                          READY     REASON
route.serving.knative.dev/helloworld-ruby   http://helloworld-ruby.default.example.com   Unknown   IngressNotConfigured

NAME                                                LATESTCREATED           LATESTREADY             READY   REASON
configuration.serving.knative.dev/helloworld-ruby   helloworld-ruby-f8cdg   helloworld-ruby-f8cdg   True

microk8s inspect

$ microk8s inspect
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/1632/inspection-report-20200826_201645.tar.gz

inspection-report-20200826_201645.tar.gz

AlexandreOuellet commented 4 years ago

I seem to be getting the same 2 errors as @ayqazi. At first, it was the "RevisionMissing" error, which is resolved by running those 2 commands (Taken from #784 ): 

sudo apparmor_parser -R /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-containerd
sudo apparmor_parser -a /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-containerd

After that, I get the "IngressNotConfigured" error, which I find very hard to fix. I've noticed that the knative-serving controller log has something to do with external ip/Load Balancing IP in its logs : 

{"level":"warn","ts":"2020-10-21T09:49:07.695Z","logger":"controller.route-controller","caller":"route/reconcile_resources.go:159","msg":"Failed to update k8s service: failed to fetch loadbalancer domain/IP from ingress status","commit":"804b3bb","knative.dev/controller":"route-controller","knative.dev/traceid":"256aa926-0dd3-4384-b28a-9cff24574da9","knative.dev/key":"default/helloworld-python"}

According to knative's documentation, I should verify that I have an external IP, and I do : 

kubectl get svc -n istio-system istio-ingressgateway
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                                                                                                                      AGE
istio-ingressgateway   LoadBalancer   10.152.183.96   192.168.1.150   15020:30671/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31580/TCP,15030:31670/TCP,15031:31009/TCP,15032:31585/TCP,15443:30433/TCP   4h3m

That IP was configured through MetalLB.

Any help would be greatly appreciated

giner commented 4 years ago

@AlexandreOuellet, you may want to try 1.19/candidate which worked for me. 1.19/stable doesn't have the fix yet, should come with the update to 1.19.3 (according to @ktsakalozos).

AlexandreOuellet commented 4 years ago

@giner I get the same error actually. I did : sudo snap refresh microk8s --channel=1.19/candidate --classic and it installed, I then deleted the knative service (the helloworld-python), then reinstalled, and same result.

AlexandreOuellet commented 4 years ago

Also, not sure how relevent this is to all this (I don't want this issue to morph), but there seems to be no istio gateway configured at all on my cluster.

According to this doc from knative, there should be an istio gateway service called knative-ingress-gateway, but when I run the command https://knative.dev/docs/serving/setting-up-custom-ingress-gateway/, there is no resources found. I believe that might explain why we're not seeing any update on the ingress, as there is no ingress handler configured.

Also of note, there is no config-istio configmap anywhere in my cluster.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.