search

rancher / os

Tiny Linux distro that runs the entire OS as Docker containers
https://rancher.com/docs/os/v1.x/en/
Apache License 2.0
6.45k stars 657 forks source link

Raspberry Pi 2 Support #417

Closed deniseschannon closed 8 years ago

faddat commented 9 years ago

When this work begins, please loop me in. I'd like to get it running in paralell on BananaPi, OrangePi, or similar.

imikushin commented 9 years ago

Sure thing. How many kinds of "Pi" are there?

faddat commented 9 years ago

If you expand your scope to Asia, you've got:

Banana Orange T-firefly

And many others even. Banana pi caught on big but I think I will be building our cheap-ass cloud from orangePi Mini 2's, they cost $25/ea and have a quad core cpu and 1gb RAM. Tbh, MetaDev is on a 6 month timeline to tear off more ports (okay all of them except a male cat5) and support PoE with 2gb RAM and a similar (if not the same) CPU.

The world needs servers! On Jul 17, 2015 8:48 PM, "Ivan Mikushin" notifications@github.com wrote:

Sure thing. How many kinds of "Pi" are there?

— Reply to this email directly or view it on GitHub https://github.com/rancher/os/issues/417#issuecomment-122278953.

luxas commented 9 years ago

I'm very interested in this. Sure there should be an ARM port. +1 for that.

I may test out some betas if it comes some. The thing I think is the difference (haven't tried banana pi) is the bootloader (the first partition of the sd card) that should be different from board to board.

I have a script for writing Arch Linux ARM, which I am using for now to SD card. The process could be something like that. The gist is here.

See also #380.

luxas commented 9 years ago

Now things are happening :smile: I've read lots of Rancher OS code and built a custom OS for me via buildroot. That way I could build my own kernel.img (is it something like vmlinuz?)

But, we face some challenges.

If I understand the RancherOS code right, the iso contains the gzipped kernel and an init ram disk. On boot, the init ram disk, does what? Does it invoke ros? This is the line that makes me think that

ln -s usr/bin/ros          ${INITRD_DIR}/init

Anyway, is the whole os run in RAM? How is docker invoked?

Over to the RPi implementation... A RPi OS is usually a .img file with two partitions, a /boot partition and a rootfs partition, as you probably know. The buildroot process gave me some required files like bootcode.bin, config.txt, cmdline.txt etc. And I think that I've read that the boot partition (which is fat formatted) must be 100M.

There is, however, a possibility that initrd works on the RPi, according to this thread, but it may be buggy...

So, which is the best way of doing this? Is it to try to use initrd or should a small busybox, buildrooted OS with ros be used or what do you think?

BTW, I did compile ros for the Pi. I got one error: #520, but I patched that and will upload it to my fork soon.

WDYT @faddat @imikushin ?

dbones commented 9 years ago

+1 for Banana as well as the rpi (1 and 2)

also are you looking to support https://www.scaleway.com/ and arm bare metal IaaS

luxas commented 8 years ago

@imikushin A week ago I was able to compile all os-images on the Pi.

I uploaded a new buildroot-config for os-base, just replaced the amd64 things with arm equalients. That code is uploaded to luxas/os-base in the raspberrypi branch.

Then I downloaded os-images to the Pi 2 and replaced the debian base image in 10-debianconsole to my own raspbian image. I removed the ubuntu console as I have no need for that one.

@dbones Scaleway would be awesome! :smile:

ibuildthecloud commented 8 years ago

@luxas Woo hoo! Glad to see progress. I'll check it out.

faddat commented 8 years ago

https://github.com/rancher/rancher/issues/2101

faddat commented 8 years ago

@luxas checking it out now, and thank you!

luxas commented 8 years ago

@faddat Just now I am compiling docker-from-scratch, as it's required for RancherOS.

Check out my Kubernetes project also. I'm trying to make RancherOS run on a Pi and then run Kubernetes on top of that...

I've compiled ros for Raspberry Pi also. PR #559

I'll publish a rancher-on-arm repo, there my scripts and such can be found.

It would be fun if someone could take a look and test this with me.

noahwhite commented 8 years ago

@luxas I'd be interested in testing that when you have it ready.

DieterReuter commented 8 years ago

@luxas Just ping me, as soon as you're ready and I'm happy to test it on some different models of Pi's (2 B, B+, B, A).

luxas commented 8 years ago

I haven't compiled the whole os, but parts of it. Status:

I'm working on publishing my scripts, so you may reproduce. This is my plan. Any suggestions?

luxas commented 8 years ago

Progress with docker-from-scratch.

Uploaded to https://github.com/luxas/docker-from-scratch. One of the issues was that the daemon returned only file not found. After some digging in the Go code, I found that it tried to mount /sys/fs/cgroup/memory which isn't present at least in Arch Linux ARM kernel.

Now, I'm trying to build whole docker statically. make cross on a amd64 machine, did only compile docker client for arm.

How do one compile the server also? @DieterReuter

DieterReuter commented 8 years ago

@luxas I'm building the Docker engine on a Pi 2 or a Scaleway ARMv7 server using this repo https://github.com/hypriot/rpi-docker-builder. But you can also try to use our pre-built binaries, which are packed in a .deb package. Here is the download site http://blog.hypriot.com/downloads/, there is a Docker engine 1.8.2 built for ARMv6.

luxas commented 8 years ago

Yeah, I'm using it atm in my docker-from-scratch, but it isn't compiled statically, unfortunately.

I saw your work earlier today with rpi-docker-builder and are wondering if it could be used for compiling statically.

ibuildthecloud commented 8 years ago

@luxas I've been playing with ARM support over the weekend and also hit the road block that we need a statically linked Docker. Let me know if you make progress on modifying rpi-docker-builder.

@luxas I haven't actually booted anything on a RPi yet (just testing with qemu atm), but you mentioned RPi kernel needs CONFIG_BLK_DEV_INITRD. That option is just for initrd support, right? Is it desirable to not boot from an initrd for these ARM devices? It is possible for us to remove the requirement for initrd, we just have to reconsider upgrades a bit.

@DieterReuter is would be great if not only was a deb available but just the statically compiled ARM binary. Similar to https://get.docker.com/builds/Linux/x86_64/docker-latest.

ibuildthecloud commented 8 years ago

@DieterReuter I say that only because I think it would be great if hypriot was the definitive community source for Docker on ARM. From that perspective not everyone is running Debian or Raspberry Pi. There's a possibility that I'll be looking at ARM64 soon.

luxas commented 8 years ago

@ibuildthecloud Yeah, it's for initrd support. They have succeeded to run initrd here, so we could make a try. Kernel support for lzma should also be present, and the kernel should be compressed, obviously, with lzma.

What other options do we have, instead of initrd? We won't package it as an .iso anyway.

DieterReuter commented 8 years ago

@ibuildthecloud I could share the binaries directly. But as you probably know it's easy to unpack a .deb with ar and tar to get the included files. ;-) I don't have time now to work on a statically linked Docker binary, but would love to support you to get it done.

ibuildthecloud commented 8 years ago

@DieterReuter statically linked is a pretty hard requirement for us, so we will for sure do our best to contribute something back. Thanks for everything you've done so far!

ibuildthecloud commented 8 years ago

@luxas The theory is if you extract the initrd to a filesystem and not have a partition with the label=RANCHER_STATE, you should be able to just boot from disk without an initrd. I say theory because I haven't really tried it. But we could get that to work if there is some bug.

DieterReuter commented 8 years ago

@ibuildthecloud I can clearly understand. Maybe we could schedule a Skype call this week (Thuesday through Thursday), getting hands on the build steps on my build server and try to get it done together? Within one or two hours we should be ready or we know what to do next!

DieterReuter commented 8 years ago

@luxas, @ibuildthecloud Here are the currently built Docker binaries for ARMv6 for direct download:

wget https://downloads.hypriot.com/docker-hypriot-1.8.2--1-armhf.tar.gz.sha256
wget https://downloads.hypriot.com/docker-hypriot-1.8.2--1-armhf.tar.gz

These binaries also running on different other ARM systems as well. Scaleway using it too on their ARMv7 cloud servers.

Building this binaries statically should be easily achieved, maybe within a few hours where I need your help!

luxas commented 8 years ago

@ibuildthecloud If we could make a statically linked docker that is uploaded to hypriot, my docker-from-scratch patches will (probably) work. What is the purpose of having an initrd for RancherOS? Maybe we could try without it first.

@DieterReuter Yeah, I'm using that link in my docker-from-scratch which I uploaded yesterday. But right now it's (of course) returning your dynamically linked docker.

Are you going to test to build docker statically now? I tried to

git clone <docker>
git checkout v1.7.1
apt-get install btrfs-tools libsqlite3-dev libdevmapper-dev fakeroot (that was in your Dockerfile)
GOARM=6 hack/make.sh binary

And I got this error:

/usr/bin/ld: cannot find -ldevmapper
/usr/bin/ld: cannot find -ldevmapper
collect2: error: ld returned 1 exit status
/goroot/pkg/tool/linux_arm/5l: running gcc failed: unsuccessful exit status 0x100

Should some devmapper library also be downloaded?

luxas commented 8 years ago

@faddat How is it going for you? Your project sounds interesting. Could you tell a little bit more about it?

DieterReuter commented 8 years ago

@luxas here is a statically linked Docker binary, but I just compiled it for ARMv7 which is for the Raspberry Pi 2 B.

wget https://downloads.hypriot.com/docker-1.8.2-static-amrv7.tar.gz
wget https://downloads.hypriot.com/docker-1.8.2-static-amrv7.tar.gz.sha256

I didn't tested it (I don't have my Pi 2 here with me), so please use it with care and give me some feedback.

luxas commented 8 years ago

@DieterReuter Thank you! The daemon is starting. Will try to run some containers on it now. Only error so far is the spelling of armv7 :smile: Could you upload the code to rpi-docker-builder? Well done!

DieterReuter commented 8 years ago

@luxas Good catch! I'm so sorry about the typo! :smirk: Right now it was just a fast quick hack and have to clean it up before I can publish it. I'm on a business trip for the next 7 days, so it will take a while.

faddat commented 8 years ago

Sure!

Caveats:

I believe that open hardware and software are key to our achievement of a star trek like future with opportunity for all. I also believe that the next few years are pivotal-- the direction that we take technology in will set the pace for future tech development, and closed tech is, by its very nature, exclusive. Think Elysium type dystopia scenarios. It all starts in August, 2012, when I leave Buffalo, NY to teach History in Taicang, Jiangsu, China. I was lucky enough to be teaching at a really great school in a really great city, at a really great time for China, despite their unfortunate firewall. I was tasked with something really cool, too: creating tech-infused history lessons. I imagine that many people reading this are fully aware of the truly great things being done with tech in the classroom, especially with regard to self-directed learning, something I'm a big believer in. All of my students had iPads, and by January 2013, I'd noticed something troubling: you can learn almost zero about computing from an iPad, and in fact it closes (clearly bad) and oversimplifies (weather this is bad or good is debatable) the device to the point that it ceases to be something that a learner can obtain intellectual ownership of while transmitting to the learner closed technological concepts & memes.

By January 2014, I'd come up with some sort of a solution to this, though it was not actionable. Due to the GFW, I had been forced into learning a great deal about

In July 2014, I left my job in Taicang to try to make my solution actionable. Basically, I wanted to take one of the mediatek chipsets used in Chinese phones/tablets and make it into a module that could be used in any computing device, and so I moved to Shenzhen. There I tried to have my first foray into selling things online, and was severely limited by the GFW. Despite that limitation I met some incredible people, including Gouwa, the guy who makes that board in the photos. I told him all about the modular system I wanted to build, probably without stopping to breathe.

I'd decided that it couldn't be an educational device as that would slur its future, limit its market and ensure that both young and old alike hated using it. Instead, it was to be general purpose computation + wireless on the smallest possible module. Servers were not as on my mind at the time, I was heading in the direction of cell phones and tablets, since that was how I'd observed young people interacting with technology. I didn't know anything would ever come of it, but it remains my chief ambition-- to get some maker of processors & radios, such as:

The concept of these phones/tablets/displays/servers (those are the original products I had in mind, but certainly there was no cluster...) was that you could swap out the "brain"-- take the processing gear out of your display (think: monitor or TV) and put it in your phone, move your phone to your tablet, and upgrade-- even from ARM to x86, etc. This is because I plan that the connectors on the modules will only carry connections out of the module, instead of a bus design like Google's ARA uses. It would recognize that it was in a phone body and change the UX to fit. Cross (that's the name of the board) is the first realization of the string of concepts surrounding Dawn, which is hopefully going to be my high-quality, mid-price open source hardware brand. Realizing that vision is a long way off, though.

In September 2014, I decided to move again, this time to Sai Gon in Viet Nam. Honestly, I accomplished little in the time that I was in Sai Gon, because of my own poor networking and lack of any context for what moving to Sai Gon entailed. Sai Gon eventually led to Da Nang, and Da Nang lasted only 3 days because I got an opportunity to work on a mobile phone project in China, which led me to head towards Ha Noi for a visa to go to China. While waiting for the visa the project was canceled, but I'd met some incredible people on my journey, who have added a tremendous amount of energy and quite essentially-- financing to this project. Seeing that I'd found a patron and he'd provided me with an incredibly large space (2500 sqm, now named MetaDev, at 672 Quang Trung in Ha Noi and BTW, 100% free if you're working on an Open Source project...) in which to pursue my projects, I decided to stay in Ha Noi. One day, not long after I showed him Rancher we hatched up a plan together, and lo and behold:

Enter: Klouds....

So those devices I'd mentioned, they need software and cloud support / webware, right? Well, they did and I'd been researching Docker in February 2015. I think this was around the time I first began to play with Rancher. Klouds.io is nearly finished, and it's going to be the first complete product from this bulwark of work I've been describing here. It is a vending machine for open source web applications that lives in the cloud, and it operates with a twist. I think that these days, most coders would like to write OSS, but are limited by organizational fear and doubt and/or a lack of funding for OSS. Klouds is automated monetization for their projects. Authors of open source webware will get ~1/3rd of the money paid by customers to help them pursue their goals. I'd like to make open source projects a sure bet for generating income, even wealth. I'd thought the site would be set on 10/1, but have had to delay till 10/10. When it goes up it will merely launch containers, and we'll be adding stuff like billing in the following two weeks-- the first two are to test the stability of the cluster and ensure that when we bill people, we're not billing them for something that isn't stable&awesome.

Okay, so you asked about the boards, the cluster boards!

I visited China a few weeks ago and of course paid a visit to Gouwa's office. When I arrived, he showed me the board I sent you the photo of. Seems he'd been listening quite carefully indeed when I spilled my guts. He's the chief bizgeek at SZWesion (founder, what have you...) and they're basically tops in my opinion when it comes to RockChip ODMs. I'll be talking with him later today and will be able to get info on the schedule for the product. We're also talking about SoFIA devices, as well. Anyway since I have been working on cloud stuff, when I saw these boards and their benchmarks, I realized immediately that these were quite possibly the disruptive kind of competitive for web servers. Thus, enter:

hurd = RancherOS + Rancher + Klouds + Cross + (cow-naming-convention)

That's what I'm going to be calling clusters of 5 or 10 4GB 8-core modules connected through a motherboard of sorts. Since OSS is awesome, and you're awesome, it looks like hurd will run your port of RancherOS. The goal is to drive down the cost and knowledge required to use highly reliable server clusters on a personal and small to medium enterprise level. We hope to migrate all of our application hosting at Klouds onto hurd clusters, as our testing shows pretty conclusively that SBC ARM servers will save a ton of money in the long run and enable us to pursue the goal of building out a global hyper-local cloud starting in markets not currently served by cloud providers today. A Hurd should be a bit smaller than a toaster. Klouds.io's cloud will enable us to host the store & other sites needed to sell Hurd and the other Dawn hardware, once they're ready.

If you read this far, then thanks! And remember: When you open source something, you're doing something far bigger than writing code or giving away hardware schematics-- you're contributing to global human prosperity and opportunity!

-Jake

On 10/5/15 5:36 PM, Lucas Käldström wrote:

@faddat https://github.com/faddat How is it going for you? Your project sounds interesting. Could you tell a little bit more about it?

— Reply to this email directly or view it on GitHub https://github.com/rancher/os/issues/417#issuecomment-145491603.

DieterReuter commented 8 years ago

@faddat thanks for this quick intro of your project, sounds really great and interesting!

faddat commented 8 years ago

By the way, the tech specs I forgot while I was recounting the whole thing are:

Cross is...

therefore Hurd is:

-Jake

On 10/6/15 4:09 AM, Dieter Reuter wrote:

@faddat https://github.com/faddat thanks for this quick intro of your project, sounds really great and interesting!

— Reply to this email directly or view it on GitHub https://github.com/rancher/os/issues/417#issuecomment-145669197.

MathiasRenner commented 8 years ago

@faddat Thanks for the insights of your project! I agree that low cost, commodity hardware powered by ARM and OSS is the path to follow.

Apart from your project description I give you a huge thumbs up for your final paragraph: When you open source something, you're doing something far bigger than writing code or giving away hardware schematics-- you're contributing to global human prosperity and opportunity!

luxas commented 8 years ago

Thank you a lot @faddat!

Some cool things about ARM boards

So, if I understand you right:

Where may one get specs of Rockchip 3368? Is this right?

I ask this because if it's going to be open source, it doesn't harm to share ideas and plans. Is there any place where one could continue this discussion? E.g. faddat/hurd?

Don't forget to run Kubernetes on top of your cluster :smile:

@ibuildthecloud Now the docker-from-scratch image is working and published! Check it out Next thing could be to try to make a .img with whole RancherOS (no initrd for this test). Which files should be included?

luxas commented 8 years ago

@ibuildthecloud @deniseschannon @imikushin Someone going to dive into this?

The next thing to do is build the RancherOS kernel, but I don't know how which files and settings that should be included.

DieterReuter commented 8 years ago

@luxas just consider to compile things on a faster Scaleway machine. It's an ARMv7 processor (4x core and 2GByte memory) and has a really fast internet connection - all just for 3 Euro per month. We're compiling Docker and building all our Docker images on it. And even the GOLANG project uses 50x machines for testing GO on ARM.

luxas commented 8 years ago

@DieterReuter Yeah, I've been considering it, maybe I'll sign up now. Is there good APIs linking e.g. Github? At least I could run my kubernetes-on-arm tests on it. That's a good idea. :smile: Hadn't thought of it that way

BTW, Have you released the scripts for building docker daemon statically?

DieterReuter commented 8 years ago

@luxas sorry, but here is the script. It's too easy, right? Works with 1.8.3 too, and for 1.9.0-dev you can skip lines 3+4.

$ git clone -b 'v1.8.2' --single-branch https://github.com/docker/docker.git
$ cd docker
$ mv vendor/src/github.com/opencontainers/runc/libcontainer/seccomp/{jump_amd64.go,jump_linux.go}
$ sed -i 's/,amd64//' vendor/src/github.com/opencontainers/runc/libcontainer/seccomp/jump_linux.go
$ curl -L https://github.com/umiddelb/armhf/raw/master/Dockerfile.armv7 > Dockerfile
$ sudo make build
$ sudo make binary
luxas commented 8 years ago

Thanks!

philipz commented 8 years ago

+1

Leopere commented 8 years ago

Whats the status on this? I would Love to try this out right now I'm on Hypriot's docker os.

ekristen commented 8 years ago

I'm very interested in the status of this as well? Where are you all with it? I wouldn't mind helping. Thanks.

lavvy commented 8 years ago

Me too. How far have we come? Would love to try On Dec 14, 2015 4:43 AM, "Erik Kristensen" notifications@github.com wrote:

I'm very interested in the status of this as well? Where are you all with it? I wouldn't mind helping. Thanks.

— Reply to this email directly or view it on GitHub https://github.com/rancher/os/issues/417#issuecomment-164334162.

luxas commented 8 years ago

Yes, thanks for the feedback I've done the initial work for this port, however I need help from the core RancherOS maintainers. @ibuildthecloud @deniseschannon @imikushin

I do not know which kernel features and libs that should be present. Check out the rest of my RancherOS ports in my profile: @luxas, my work is in raspberrypi branches. But, they have to be rebased when the maintainers decide to continue to support this work.

Anyway, our maintainers are doing excellent work anyway and this isn't a high priority thing.

ekristen commented 8 years ago

@luxas can you point out which repo's specifically? I don't see a raspberrypi branch in the os fork.

ekristen commented 8 years ago

Nevermind, found it. For others https://github.com/luxas/os-base/tree/raspberrypi

luxas commented 8 years ago

And https://github.com/luxas/os-images/tree/raspberrypi https://github.com/luxas/docker-from-scratch/tree/raspberrypi

In os I made a change so it worked on 32-bit machines, but that PR is already merged. So no unmerged things in os at this time

tfindlay-au commented 8 years ago

... now to adapt this for Cubox (4Gb ram + Freescale iMX6 (4 ARM A9 cores)).

Thanks for sharing!

ibuildthecloud commented 8 years ago

If you run the build from https://github.com/rancher/os/pull/735 (follow the directions in the PR), it will produce an initrd that can be used on RPi 2 and probably easily adapted to most armv7 and newer board. You don't actually boot with the initrd but instead extract the contents of it to the root filesystem from which you will be booting. Just make sure you add init=/init rw to the kernel cmdline for whatever your booting. We really hope to have RPi2 support merged in the next 2 weeks and then we will move on to supporting more ARM board, specifically some ARM64 ones.