Installation issues with latest hypriot-os for rpi3

[richtong]

I don't know if ur still maintaining this but rpi3 on Hypriot requires a different download and rpi2 and rpi1 have newer images. Or should we migrate to k8s 1.3-alpha2?

[luxas]

Hi @richtong! Yes, I'm still maintaining actively. I've taken incredible steps forward together with others in the Kubernetes team the last weeks. Example

The long term goal for this project is to make sure ARM is a first-class citizen in the Kubernetes world, as it boosts education and lets everybody test it out easily

I've noticed the Hypriot bump needed, sorry for not getting it into v0.7.0, but I didn't own a RPi 3 to test on back then. However, I've added it to dev more than a week ago.

I haven't really made up my mind yet if an unstable release (v1.3.0-alpha.x) should be present in v0.7.5. What do you think?

[richtong]

You are doing amazing work, kudos to you Lucas. it is not easy to integrate into a big team like that. Personally, I think it would really enhance testing for people who arm folks. After all, you have to have an underlying operating system. Nice to have one preferred way and your repo is the one that integrate hypriot and kubernetes. :-)

Thanks Rich

—

Everything will be all right in the end and if it's not all right, then it's not yet the end. Sonny Kapur. The Best Exotic Marigold Hotel, 2012.

tongfamily.com | fb.me/richtong | @richtong | +1-206-963-9029 | +86-137-6184-9029

On Apr 20, 2016, at 6:53 AM, Lucas Käldström notifications@github.com wrote:

Hi @richtong https://github.com/richtong! Yes, I'm still maintaining actively. I've taken incredible steps forward together with others in the Kubernetes team the last weeks. Example https://github.com/kubernetes/kubernetes.github.io/pull/407 The long term goal for this project is to make sure ARM is a first-class citizen in the Kubernetes world, as it boosts education and lets everybody test it out easily

I've noticed the Hypriot bump needed, sorry for not getting it into v0.7.0, but I didn't own a RPi 3 to test on back then. However, I've added it to dev more than a week ago.

I haven't really made up my mind yet if an unstable release (v1.3.0-alpha.x) should be present in v0.7.5. What do you think?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/luxas/kubernetes-on-arm/issues/95#issuecomment-212434284

[richtong]

Also, how do I get the rpi3 version. I followed your instructions (way clearer than the kubernetes for docker by the way!) but do not see the rpi3 option.

Also, is there an easy way to just run kubectl install from the command line, we are bringing up a 40 node cluster and while I could hack away with echo, it would be nice to just set some parameters. I could modify kube-config (not kubectl sorry for the confusion) to accept flags as well and give you a PR too if that is easier.

[richtong]

Finally, is thee some place where the v.1.3.0-alpha on rpi is documented? I can't seem to find and am shooting the dark trying to modify command lines from the kubernetes on docker http://kubernetes.io/docs/getting-started-guides/docker/#download-kubectl. the line gcr.io/google_containers/hyperkube-amd64:${K8S_VERSION} in the docker run command seems the most wrong. but how do I see what the arm version is named?

[richtong]

Note that for the rpi3 on the latest hypriot version, they switched to device-init, so you have go edit /boot/device-init.yaml instead of /boot/occidentalis.txt on the rpi1 and rpi2 versions of hypriot. They also changed account names and passwords.

[luxas]

40 node cluster

Wow! Nice to have an impact on a such project :) Normally, we could've chatted on Slack or something other in more detail to improve my docs, but I'm going on a one-week trip from monday.

To download kubectl, just do this:

curl -sSL http://storage.googleapis.com/kubernetes-release/release/v1.3.0-alpha.2/bin/linux/arm/kubectl > /usr/bin/kubectl
chmod +x /usr/bin/kubectl

Here's docs for the new official and experimental ARM images: https://github.com/kubernetes/kubernetes.github.io/pull/407

To run the latest Kubernetes release locally on a Raspberry Pi is dead simple thanks to all my PRs (here's the tracking issue) for main k8s:

export ARCH=arm
export K8S_VERSION=v1.3.0-alpha.2
docker run -d \
    --volume=/:/rootfs:ro \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:rw \
    --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
    --volume=/var/run:/var/run:rw \
    --net=host \
    --pid=host \
    --privileged=true \
    gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \
    /hyperkube kubelet \
        --containerized \
        --hostname-override="127.0.0.1" \
        --api-servers=http://localhost:8080 \
        --config=/etc/kubernetes/manifests \
        --cluster-dns=10.0.0.10 \
        --cluster-domain=cluster.local \
        --allow-privileged=true --v=2

However, remember that that's only for a one-node cluster. If you want more nodes, use Kubernetes on ARM :smile: (Of course it's possible to use hyperkube-arm for multinode clusters too, but then you have to know what you're doing: http://kubernetes.io/docs/getting-started-guides/docker-multinode/)

Also, is there an easy way to just run kubectl install from the command line,

There is no such thing as kubectl install. kubectl is the Kubernetes client that talks to the apiserver. Maybe you mean kube-config install? kube-config is my bash script for automating the Kubernetes installation on the Pis, and has absolutely nothing with official Kubernetes to do.

/boot/device-init.yaml instead of /boot/occidentalis.txt

Hadn't noticed that. Will update before releasing the next version. Preferably the Hypriot guys will release a "stable" (not beta) version before that too.

Lastly, if you want to write a Hypriot v0.7.0 card with Kubernetes on ARM v0.7.0 a) Apply this commit locally: 17b9b36900049071eb46a990c8177c1a7f395682 b) Use the .deb deployment on an already-flashed HypriotOS installation.

[richtong]

Have fun on your trip. Yes what slack forum are u using for this. I'm bringing up nodes as fast as I can and your work is just incredible. We are doing some cool stuff with computer vision and machine learning so you already supports a 800 megapixel camera array!

Thanks, Rich

No! Try not. Do, or do not. There is no try. Master Yoda. Star Wars Episode V: The Empire Strikes Back

http://tongfamily.com | @richtong | http://fb.me/richtong1 | +1-206-963-9029

Sent from my iPhone

On Apr 23, 2016, at 13:44, Lucas Käldström notifications@github.com wrote:

40 node cluster

Wow! Nice to have an impact on a such project :) Normally, we could've chatted on Slack or something other in more detail to improve my docs, but I'm going on a one-week trip from tomorrow.

To download kubectl, just do this:

curl -sSL http://storage.googleapis.com/kubernetes-release/release/v1.3.0-alpha.2/bin/linux/arm/kubectl > /usr/bin/kubectl chmod +x /usr/bin/kubectl Here's docs for the new official and experimental ARM images: kubernetes/kubernetes.github.io#407

To run the latest Kubernetes release locally on a Raspberry Pi is dead simple thanks to all my PRs (here's the tracking issue) for main k8s:

export ARCH=amd64 export K8S_VERSION=v1.3.0-alpha.2 docker run -d \ --volume=/:/rootfs:ro \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:rw \ --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \ --volume=/var/run:/var/run:rw \ --net=host \ --pid=host \ --privileged=true \ gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \ /hyperkube kubelet \ --containerized \ --hostname-override="127.0.0.1" \ --api-servers=http://localhost:8080 \ --config=/etc/kubernetes/manifests \ --cluster-dns=10.0.0.10 \ --cluster-domain=cluster.local \ --allow-privileged=true --v=2 However, remember that that's only for a one-node cluster. If you want more nodes, use Kubernetes on ARM (Of course it's possible to use hyperkube-arm for multinode clusters too, but then you have to know what you're doing: http://kubernetes.io/docs/getting-started-guides/docker-multinode/)

Also, is there an easy way to just run kubectl install from the command line,

There is no such thing as kubectl install. kubectl is the Kubernetes client that talks to the apiserver. Maybe you mean kube-config install? kube-config is my bash script for automating the Kubernetes installation on the Pis, and has absolutely nothing with official Kubernetes to do.

/boot/device-init.yaml instead of /boot/occidentalis.txt

Hadn't noticed that. Will update before releasing the next version. Preferably the Hypriot guys will release a "stable" (not beta) version before that too.

Lastly, if you want to write a Hypriot v0.7.0 card with Kubernetes on ARM v0.7.0 a) Apply this commit locally: 17b9b36 b) Use the .deb deployment on an already-flashed HypriotOS installation.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[richtong]

Yes u haven't noticed I think because they don't officially support the rpi3 in their version of stable. They have beta builds which I'm using (they r fantastic guys too!).

Sorry I got your script confused with the kubectl app. We are building automated install of course as a pre bootstrap before docker.

Thanks, Rich

No! Try not. Do, or do not. There is no try. Master Yoda. Star Wars Episode V: The Empire Strikes Back

http://tongfamily.com | @richtong | http://fb.me/richtong1 | +1-206-963-9029

Sent from my iPhone

On Apr 23, 2016, at 13:44, Lucas Käldström notifications@github.com wrote:

40 node cluster

Wow! Nice to have an impact on a such project :) Normally, we could've chatted on Slack or something other in more detail to improve my docs, but I'm going on a one-week trip from tomorrow.

To download kubectl, just do this:

curl -sSL http://storage.googleapis.com/kubernetes-release/release/v1.3.0-alpha.2/bin/linux/arm/kubectl > /usr/bin/kubectl chmod +x /usr/bin/kubectl Here's docs for the new official and experimental ARM images: kubernetes/kubernetes.github.io#407

To run the latest Kubernetes release locally on a Raspberry Pi is dead simple thanks to all my PRs (here's the tracking issue) for main k8s:

export ARCH=amd64 export K8S_VERSION=v1.3.0-alpha.2 docker run -d \ --volume=/:/rootfs:ro \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:rw \ --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \ --volume=/var/run:/var/run:rw \ --net=host \ --pid=host \ --privileged=true \ gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \ /hyperkube kubelet \ --containerized \ --hostname-override="127.0.0.1" \ --api-servers=http://localhost:8080 \ --config=/etc/kubernetes/manifests \ --cluster-dns=10.0.0.10 \ --cluster-domain=cluster.local \ --allow-privileged=true --v=2 However, remember that that's only for a one-node cluster. If you want more nodes, use Kubernetes on ARM (Of course it's possible to use hyperkube-arm for multinode clusters too, but then you have to know what you're doing: http://kubernetes.io/docs/getting-started-guides/docker-multinode/)

Also, is there an easy way to just run kubectl install from the command line,

There is no such thing as kubectl install. kubectl is the Kubernetes client that talks to the apiserver. Maybe you mean kube-config install? kube-config is my bash script for automating the Kubernetes installation on the Pis, and has absolutely nothing with official Kubernetes to do.

/boot/device-init.yaml instead of /boot/occidentalis.txt

Hadn't noticed that. Will update before releasing the next version. Preferably the Hypriot guys will release a "stable" (not beta) version before that too.

Lastly, if you want to write a Hypriot v0.7.0 card with Kubernetes on ARM v0.7.0 a) Apply this commit locally: 17b9b36 b) Use the .deb deployment on an already-flashed HypriotOS installation.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[luxas]

@richtong

(they r fantastic guys too!).

Totally agree, but in this case I and the guy that made alpha builds for ARM is the same person :smile: Kubernetes will not "support" ARM commercially, but binaries are uploaded to http://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/linux/arm/${K8S_BINARY} since v1.2.0-alpha.8 and hyperkube images are built since v1.3.0-alpha.2

That's my long-term goal with this repository, to provide Kubernetes on ARM; an easy out-of-the-box solution and at the same time merge my code with core Kubernetes so even more people may use it.

Nowadays, I'm a Kubernetes maintainer, and my largest assignment is https://github.com/kubernetes/kubernetes/issues/17981. From there you'll see links to everything that's happening in this area.

Yes what slack forum are u using for this.

http://kubernetes.slack.com One may chat privately there too, if the matter is something that's outside core discussions.

[richtong]

Well it is nice to be 2x a great person! Wow that is a lot of work. Will work on getting the cluster working and report back to you. I think there is a slight typo in your previous comment the VERSION should be arm and not amd64, but I get the idea. Will try on RPI1 and RPI3. Also joined kubernetes.slack.com as richt

Thanks Rich

---

To get down the hill, you just have to turn, just keep turning, turning, turning. Calvin Tong. Whistler-Blackcomb Mountain, 2009

tongfamily.com | fb.me/richtong1 | @richtong | +1-206-963-9029

On Apr 24, 2016, at 8:50 AM, Lucas Käldström notifications@github.com wrote:

@richtong https://github.com/richtong (they r fantastic guys too!).

Totally agree, but in this case I and the guy that made alpha builds for ARM is the same person Kubernetes will not "support" ARM commercially, but binaries are uploaded to http://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/linux/arm/${K8S_BINARY} since v1.2.0-alpha.8 and hyperkube images are built since v1.3.0-alpha.2

That's my long-term goal with this repository, to provide Kubernetes on ARM; an easy out-of-the-box solution and at the same time merge my code with core Kubernetes so even more people may use it.

Nowadays, I'm a Kubernetes maintainer, and my largest assignment is kubernetes/kubernetes#17981 https://github.com/kubernetes/kubernetes/issues/17981. From there you'll see links to everything that's happening in this area.

Yes what slack forum are u using for this.

http://kubernetes.slack.com http://kubernetes.slack.com/ One may chat privately there too, if the matter is something that's outside core discussions.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/luxas/kubernetes-on-arm/issues/95#issuecomment-213987625

[richtong]

Ok using the alpha hypriot-os that works for rpi-3, if you run your install on it, it causes the rpi to hang and ssh not to work anymore :-( Seems like there is incompatibilities between your stuff on the rpi3 and the latest hypriot betas. Not surprising, but working through debug now. It appears that it causes the rpi3 to hard crash and shutdown.

What's the best way to diagnose?

[richtong]

I'm using 0.4.9 of their latest build https://github.com/hypriot/image-builder-rpi/releases

[stevesloka]

I recently got 4 rpi3's and got everything up and running. I used Version 0.7.0 Berry (beta) of Hypriot and then did the systemd install on each pi by cloning the repo. Everything so far has been fine.

[luxas]

Thanks @stevesloka and nice to know! Are you running Ubernetes?

[stevesloka]

No not yet @luxas but my plan was to do that next. I have Ubernetes-lite running in AWS on CoreOS right now. I just need to tag instances and see what's involved in the code since I'm not running a "cloud" provider.

Also side not (and can open issue if wanted), is there an easy way to upgrade without re-flashing the cards?

[madmas]

Hi, I just reinstalled a 6 rpi3 cluster with hypriotos (hypriot-rpi-20160306-192317.img) and k8s-on-arm deb packages. Used rpi2 during kube-config install and it seems to run fine (didn't do much with it after reinstall, yet). Also enabled DNS. will do more with it soon, thank for all the work you've put in it :+1:

[luxas]

@stevesloka No, there's no really easy way of "reflashing" kubernetes-on-arm. Haven't tested reinstalling the deb package, but I think that's as close as we get right now. But it would be appreciated if you may think about a way of doing it :)

[luxas]

Closing this since v0.8.0 has an rpi-3 option