search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.42k stars 768 forks source link

microk8s installation not working on Ubuntu 20.04.1 AWS EC2 #1540

Closed uGiFarukh closed 1 year ago

uGiFarukh commented 4 years ago

ubuntu@ip-172-31-29-202:~$ sudo snap install microk8s --channel=1.19 --classic error: cannot perform the following tasks:

<exceeded maximum runtime of 5m0s> -----) ubuntu@ip-172-31-29-202:~$

t2.micro instance. tried spinning up two separate instances and the same issue.

uGiFarukh commented 4 years ago

microk8s v1.18.6 from Canonical✓ installed

So the stable channel installation is working. But somehow the latest 1.19 which I want to use is not installing. Also, I noticed that the download speed for the stable channel was much quicker compared to the 1.19 channel which was around the 300KB/s neighborhood. Is there anything particularly wrong with the 1.19 channel?

ktsakalozos commented 4 years ago

Hi @uGiFarukh

The error you see is because the API server did not manage to start within 5 minutes from the installation of the snap. A t2.micro has not enough resources (mainly RAM) to host MicroK8s and any workloads. The 1.18 deployment displays a different behavior in such cases. It will deploy and will let you debug any issues present later. We have seen for example cases where in-node processes get killed due to out of memory exceptions. Not sure what approach is better the 1.19 approach of failing the deployment if the API server will not start seems more fair to me. We are looking in ways to reduce the K8s memory requirements.

1.19 downloading at a low speed is because the CDN is not populated yet with the latest 1.19 snap revision we pushed.

uGiFarukh commented 4 years ago

Hello @ktsakalozos

I thought microk8s is designed to work on IoT ARM devices. These devices contains very low amount of RAM to work with. My application is mainly focused on IoT devices and EC2 t2.micro instances. What would you recommend the best approach for my use case? I really do wanted to use the latest 1.19 kubernetes with everything. Is there any way, I can make it work in this RAM constraint environments without breaking anything?

Also, when will the CDN be populated for the 1.19 snap revision?

ktsakalozos commented 4 years ago

Could you try adding some swap memory to your nodes? Here is how I got MicroK8s on t1.micro:

sudo fallocate -l 3G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo snap install microk8s --classic --channel=1.19
# Note you have to make swap permanent

I do not know when CDNs (or whatever mechanism is caching snaps on AWS) get updated.

uGiFarukh commented 4 years ago

@ktsakalozos

I am using the most latest ubuntu 20.04.1 LTS server on t2.micro - So, basically have 1vCPU and 1GB RAM

Can you please let me know using what sets of linux commands I can set permanent swap so that microk8s 1.19 works properly without any problem? Do I need to add anymore commands to the ones you provided on the last reply to make the 3GB swap permanent? Please let me know in detail.

Thank You.

uGiFarukh commented 4 years ago

And what should I do if I want to run microk8s 1.19 in production IoT devices which has 512MB or 1GB ram? should I add permanent swaps to them also? And will this method work properly without any problems?

ktsakalozos commented 4 years ago

To make swap permanent have a look at [2]. Also make sure you go through [1] to understand the effects of swap on kubernetes. I am afraid it is a challenge to fit the OS, kubernetes and any workload in 512MB.

[1] https://discuss.kubernetes.io/t/swap-off-why-is-it-necessary/6879 [2] https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

uGiFarukh commented 4 years ago

@ktsakalozos btw I had couple of other questions for which I thought I shouldn't open up another issue.

microk8s is a single binary kubernetes? or it uses multiple binaries to achieve different features? Also, what is the exact storage and memory footprint of microk8s? And will it run properly on ARMv6 armhf architecture? Planning to use dietpie and microk8s on top of it.

Btw, k3s by rancher labs seems to work perfectly fine on 512MB IoT systems. What are they doing differently to achieve this? Any idea?

uGiFarukh commented 4 years ago

@ktsakalozos

Well, microk8s installed on the t2.micro with the extra swap space but it is practically unusable. I tried to use the kubernetes dashboard and I think it crashed the API. Not sure how the use case of microk8s will be in IoT or embedded devices :(

ktsakalozos commented 4 years ago

microk8s is a single binary kubernetes?

No, MicroK8s is not a "single binary Kubernetes".

MicroK8s is the upstream Kubernete binaries, adds HA functionality (with dqlite) and packages all of k8s dependencies in a single package called snap [1]. This presents a few advantages that make it a great fit for IoT (and not only for IoT).

will it run properly on ARMv6 armhf architecture

The currently supported architectures are AMD64 and ARM64. There is no technical reasons why we have not a build for armhf.

What are they doing differently to achieve this?

You can get some hints on their site.

[1] https://snapcraft.io/docs [2] https://microk8s.io/docs/setting-snap-channel [3] https://kubernetes.io/docs/home/

cedricdv commented 3 years ago

Could you try adding some swap memory to your nodes? Here is how I got MicroK8s on t1.micro:

sudo fallocate -l 3G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo snap install microk8s --classic --channel=1.19
# Note you have to make swap permanent

I do not know when CDNs (or whatever mechanism is caching snaps on AWS) get updated.

This helped me alot! Was trying to get this thing going on a cluster of 7 Raspberry Pi modules and I was running into the same issue.

itamararjuan commented 3 years ago

I'm still having the same issue with Ubuntu 20 on AWS. Trying to join another node into a fresh cluster. nodes are t3-medium both of them (2 vCPUs and 4 GB of RAM if I'm not mistaken - definitely not "weak" machines) I was able to get it working when installing the 1.19 channel.

When I try to install the latest one as pointed out by the documentation to be the latest - it doesn't work and I cannot join the nodes.

Also, one more thing to note is that the documentation claims the node who issued the add-node command should automatically assume a manager role in the cluster, but it doesn't really work this way for some reason. when I query the nodes using microk8s kubectl get no

both nodes are now available but both of them are with ROLES set to <none>

Omniscience619 commented 2 years ago

@ktsakalozos

Well, microk8s installed on the t2.micro with the extra swap space but it is practically unusable. I tried to use the kubernetes dashboard and I think it crashed the API. Not sure how the use case of microk8s will be in IoT or embedded devices :(

I second this. I've been trying to run a multi-node MicroK8S cluster with swap enabled as recommended by @ktsakalozos , and even after all that, kube-api server times out and gives context deadline issues.

Moreover, the second node that I want to connect isn't able to communicate with the first one at all. Gotta get better instances I guess.

As an example, have a look at the following response.

$ microk8s kubectl get pods -n kube-system
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get pods)
stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.