search

luxas / kubernetes-on-arm

Kubernetes ported to ARM boards like Raspberry Pi.
MIT License
596 stars 86 forks source link

Odroid C2 #74

Open DorianGray opened 8 years ago

DorianGray commented 8 years ago

So, I've made a stripped down image of ubuntu(2.2gb) that works on odroid c2 and hacked the scripts here together to make it work. I was surprised at how little compilation I had to do, ubuntu's aarch64 support is stellar. I have a couple more things to do, mostly cleanup, but almost there! I'll host the image somewhere like s3, or maybe github if they'll let me once it's done(the official image's kernel is insufficient and is more than twice as big as it should be).

I'm thinking about changing out the root filesystem for btrfs and having docker just use subvolumes. Beyond that, maybe enabling compression to increase disk bandwidth. This thing has ample cpu power but not so much disk space or bandwidth. I dug into it and it looks like it will be a good chunk of work + testing to refactor things to allow for a configurable storage driver. Before I start, would that be something that you would want in the core scripts or should I just keep the main version using overlayfs?

As an aside, it still uses armhf kubernetes binaries but I built an aarch64 docker 1.10.2 using go1.6 on the machine itself. I have my cluster's master on armhf and a single aarch64 worker. I tagged the node as arch=aarch64, added a nodeSelector: arch: aarch64 for a 64bit jenkins and presto, it all worked! I guess I should have assumed cross architecture wouldn't be a problem, but I was very happy when it just worked.

luxas commented 8 years ago

Which kernel version do you have? The newer, the better. Arch Linux ARM has 3.14 Use overlayfs if possible, if not, check how I did the parallella.sh file in dynamic-env that swaps the overlay driver for something that works.

We should use armhf binaries for the time being for arm64 too, but after some releases we might use k8s arm64 ones.

DorianGray commented 8 years ago

Kernel is 3.14 with several docker specific features backported. I just had to enable a few flags extra to work with aufs and overlay...overlay is still giving me trouble, but I'll see if I can't get it working.

DorianGray commented 8 years ago

Btw, theoretically the rasp pi 3 could work on this image with some /boot modifications... they do use the same cpu.

DorianGray commented 8 years ago

I am going to have to use btrfs for performance and reliability reasons. AUFS is too slow and wonky and overlay doesn't work in this kernel...

Even after having it patched and the module loaded docker won't recognize the overlay driver. It probably has a hard check on kernel version or is looking for a slightly different name or something weird.

DorianGray commented 8 years ago

https://github.com/luxas/kubernetes-on-arm/pull/75

larmog commented 8 years ago

I got, just now, Docker docker-hypriot_1.10.3-1_armhf.deb working on C2 Odrobian (Debian based) with 32-bit (armhf) support. Maybe that's an better alternative?

DorianGray commented 8 years ago

No aarch64 support for applications though =(

luxas commented 8 years ago

Why not just Arch Linux? And I saw that you @larmog has tested out Hypriot too. Success or failure?

DorianGray commented 8 years ago

Arch linux 64 sounds like it will work. It would need to use the odroid c2 kernel and I'm not sure what, if any, uboot modifications would be required. Personally, I don't want to invest the time in it right now. It works on ubuntu now for my purposes, and I am willing to spend the time to support rancheros aarch64 in the future because I think that's the long term solution for this project.

larmog commented 8 years ago

@luxas I couldn't get the Hypriot image to boot. It hangs in U-boot. I wanted an image that supported both armhf and aarch64 (multiarch) and found Odrobian. I've verified that it runs both armhf and aarch64 images and it works with 0.7.0-dev.

Arch Linux should work, we just need to enable multiarch, until there's a 64-bit build of the binaries.

luxas commented 8 years ago

I'm working on upstream binaries for arm64. It depends though on https://github.com/kubernetes/kubernetes/pull/22149 and some other things.

If you want to try out arm64 binaries anyway, I quickly hacked a script for that: https://github.com/luxas/kubernetes-on-arm/blob/dev/scripts/build-k8s-on-amd64/Dockerfile.arm64. Haven't tested it yet.

I'll have to think about how to use/implement #75 and #79 in the best way. I want Odroid support to this project although I haven't one (yet). But I want it independently maintained like Arch Linux or HypriotOS

larmog commented 8 years ago

@luxas I reviewed the code for Arch. I didn't know you used your own static compiled version of Docker. I use hypriotos and used your kubernetes-on-arm deb package.

I can't see any problems building an image for Arch and C2 the same way. Personally I would prefer to use the package manager for handling binaries like Docker.

I think it would be beneficial (initially) to have an image that support both armhf and aarch64 in cases (like me) where you have a mixed architecture.

I'll try to add Arch support for C2, if I can find the time. I'm sure the Hypriot team will release a C2 image of hypriotos in a near future.