search

rancher / os

Tiny Linux distro that runs the entire OS as Docker containers
https://rancher.com/docs/os/v1.x/en/
Apache License 2.0
6.44k stars 655 forks source link

Improve ARM experience #760

Open luxas opened 8 years ago

luxas commented 8 years ago

This will be a tracking issue for optimizing the ARM port.

Related: #417 #735 @imikushin @ibuildthecloud

luxas commented 8 years ago

Another thing: your rancher/docker-arm release isn't runnable on ARMv6 (RPi 1), so if we want RPi 1 support, we have to build it in with e.g. resin/rpi-raspbian as base image and GOARM=6

DieterReuter commented 8 years ago

@luxas If you like, we can discuss some of these points in detail to build an optimised SD card image for the RPi2. I did already built a first version, but would like to get to know your thoughts. Currently it's only running on ARMv7, thus I removed the dual kernel support and saved some additional space. 

wget http://downloads.hypriot.com/rancheros-rpi2-20160221.zip

Current build includes:

luxas commented 8 years ago

Yeah, maybe we should target Pi 2 first (a bird whispered that a Pi 3 is coming :smile:) and then consider a separate Pi 1 support for saving space.

Have you uploaded the build script or something? Maybe we could work on a Gist?

luxas commented 8 years ago

Now rancher asks for a new password on first boot. That's good. The sudo problem I had is also fixed.

Some stats on first boot: free -m: 125 MB du -sh /: 197 MB ps aux | wc -l: 76 processes It uses pretty much 0-1% of the cores when idle

A funny thing, when running sudo system-docker ps, it says that the containers were created 46 years ago :) Probably because 1970 is the default startup time.

Will investigate even more. I got Kubernetes running on this image now :tada: Yay!

In fact, the only thing I had to change was sudo modprobe tun When running a Kubernetes master ~640 MB is consumed, so it's OK.

I'm excited to be a part of this work :) Let's move on to the script...

DieterReuter commented 8 years ago

@luxas I just used the script from @ibuildthecloud and done some optimisations. So maybe it's best, if you start with his script here https://github.com/rancher/os/tree/master/scripts/images/rpi-hypriot.

As soon as I have the time, I can cleanup my improved build script to publish it (hopefully on the weekend).

spralle commented 8 years ago

How is RPi 3 compatibility?

DieterReuter commented 8 years ago

@spralle You already have one? Honestly, we can only tell anything about the RPi 3, when we were able to test it.

luxas commented 8 years ago

I ordered mine ~20 mins ago. I expect it to ship in 2-3 days. To answer your question, yes, RancherOS is RPi compatible (at least theoretically) I'm working right now on arm64 RancherOS rootfs though.

spralle commented 8 years ago

@DieterReuter, Yes but I haven't had time to play with it yet. If I can do something to test it just point me in the right direction.

JulekFr commented 8 years ago

Hi @DieterReuter, Thank you for the img. I am just struggling on one point, I successfully resized the partition but after reboot rootfs doesn't expand. Could you explain briefly the process?

DieterReuter commented 8 years ago

@JulekFr, we do have a resizefs script, which we run in HypriotOS at first boot only. The partition table of the SD card has to be changed in order to expand the rootfs.

The same thing has to be done in RangerOS at first boot, but I think we don't have all the used Linux tools installed. So I guess, one has to modify the partition table directly and has to test this in RancherOS.

luxas commented 8 years ago

@JulekFr Here's the RancherOS builtin way of doing it: https://github.com/rancher/os-images/tree/master/src/docker/10-resizefs. There's a PR open for ARM support for that image too, and will hopefully be in for v0.4.4 or something.

luxas commented 8 years ago

@ibuildthecloud @imikushin What are your opinions about RPi 1 support? I'm leaning towards that it's not worth it really, that we should see 1024MB RAM as a hard limit. Then we could build binaries for ARMv7+ only, and focus on arm64 support for RPi 3. WDYT?

DieterReuter commented 8 years ago

@luxas According to RPi 3, I don't expect any improvement to run 64-bit on this board. And there is currently no support for 64-bit from the Raspberry Pi Foundation at all. The device is currently and will be limited to 1GByte for a longer time and in this case it's more efficient to use it with 32-bit userland binaries ONLY! With 1GByte you will have only disadvantages with 64-bit, the binaries will be 50% to 100% larger in size and consume more memory when running. Just my two cents.

luxas commented 8 years ago

Yeah, you're completely right about the RPi 3. Was just too excited about getting an ARM64 board to experiment with :smile:, so I didn't think about the 64-bit facts with larger size, RAM usage etc. compared to 32-bit ones. Thanks for pointing that out.

The RPi is stuck at 1GB RAM, because of the VC4, IIRC?

What do you think about the Pine64 with 2GB RAM? Is it worth to use 64-bit there? At least it would be cool with RancherOS for ARM64 I'm still waiting for my Pine64 to ship...

DieterReuter commented 8 years ago

@luxas, I do have a Pine A64+ with 1GByte here for testing and right now the Linux support should be there - at least for the first tests. Maybe this would be a could experiment to compare and benchmark this board between 32- and 64-bit userland. My rough guess here: 32-bit will win.

But it would be a nice and cheap testbed for 64-bit ARMv8 Docker Images.

luxas commented 8 years ago

I know this benchmark is quite silly, but it might give a hunch about what's better. https://github.com/luxas/benchmark Then one could compare if 32-bit or 64-bit compilations are faster. WDYT?

marcelhodan commented 8 years ago

@DieterReuter Have you successfully tested the Pine A64+ ? And if so, do you still have the .img file? I would really like to add my (currently not used) Pine64 to my personal Rancher environment.

noplanman commented 6 years ago

Has there been any progress on getting RancherOS running on a Pine64?

filviu commented 6 years ago

I'm chiming in for pine64 support as well. I have a nice clusterboard (7 sopine modules, each quadcore with 2GB RAM) on which I'd like to play with rancher around.

ppouliot commented 5 years ago

Is there any plan for supporting server ARM64 platforms like Ampere Computing eMAG?