Management Role includes "includes/install_k8s_resource.yml"?

[robertpeteuil]

Is it meant to point to includes/install_binaries.yml?

FYI - i love your repo. It's very clean & it helped proactively fix a lot of problems before they occurred (like the same machine_id on all nodes, etc...)

There are a few tasks that are no longer necessary with HypriotOS 1.5. Should I just list them for you, or would you like me to submit a PR?

[rhuss]

Oops, it looks like that I forgot to add the file. It's meant to be used for applying k8s resources. Just updated it.

thanks for the compliments ;-) Although I'm only able to update the repo from time to time, I'm still committed to it.

If you would have time to submit a PR, that would be awesome! Otherwise, I will update the repo to latest Hypriot and Kubernetes, but can't promise when.

[rhuss]

@robertpeteuil I'm about to update for HypriotOS 1.5 and it would be awesome if you could point me to the tasks which are obsolete now.

[robertpeteuil]

@rhuss Absolutely, I'm happy to share my findings.

First, let me thank you - your repo is great. My original plan was to fork it and submit PRs. Unfortunately, my project requires some substantial changes which made a fork impossible. But, I'll make sure to message you when my repo is posted (and give you proper attribution).

Now - here's the tasks that are obsolete, and some you may want to consider adjusting.

Revisions for HypriotOS 1.5 setup.yml playbook

• installing python no longer necessary

role: base - file "main.yml"

• may not need to add "swap" - as it's not supported by k8s (unless you have some other use)

role: base - file "apt.yml"

• don't need to add Hypriot key & repo:
  • already installed in /etc/apt/sources.list.d/hypriot.list:
  • deb https://packagecloud.io/Hypriot/rpi/debian/ jessie main
  • deb https://packagecloud.io/Hypriot/Schatzkiste/debian/ jessie main
  • misc changes:
  • HypriotOS 1.5 is Jessie based
  • many repos are already added in files in the /etc/apt/sources.list.d folder
• don't need to remove kubernetes-xenial-stable
• don't need to set overlay for docker (see docker notes below)
• optional: consider moving docker-pin & downgrade tasks to kubernetes role (see docker notes below)

role: base - file "system.yml"

• don't need to add cgroup to bootparams (it's already there)
• don't need to add overlay filesystem module (see docker notes below)
• don't need to load overlay module (see docker notes below)
• may be able to set timezone with Ansible "Timezone" module (requires a reboot to take effect)

  timezone:
  name: "{{ timezone }}"

• don't need to check/link ld-linux-armhf.so.3 (already exists)
• optional: may want to verify the dhcp tasks still work as expected
  • Only mentioned because I've not tested them because I've implemented static-IP assignment differently

role: base - template "hosts"

• may want to change {% for item in groups['pis'] %} to {% for item in ansible_play_hosts %}
  • the current version is hardcoded to the group 'pis'
  • the new version uses an Ansible variable containing a list of hosts the playbook is running against
  • this prevents the template from breaking or copying the wrong info, if the group is dynamically assigned or renamed in the future

role: base - file "user.yml"

• remove task "add user pi to group docker" on line 23, it's already done as part of the task on line 8
• optional: consider adding "video" to groups user "pi" is added to on line 8
  • checking RPi HW settings like temp or cpu/core speed required membership in "video" group

role: kubernetes - file "kubeadm.yml"

• shouldn't the check for "running cluster" be first?
  • if cluster is running and reset not requested, tasks for token check/generation/set_fact are unnecessary
• task block "create token from master" actually runs on all hosts (if the token doesn't exist)
  • this doesn't break anything, as you still end up with only one file/token
  • but the logic is somewhat broken

docker notes

• HypriotOS 1.5 contains Docker 17.05.0-ce with overlay2
  • despite my comment in a different issue, overlay2 works
  • additionally, overlay2 works after downgrading Docker (if the current overlay tasks aren't run)
  • the cause of the error I mentioned in the other issue is very specific and should not occur in normal use (I can explain why & how it happens if you want to know)
• The current script downgrades Docker to the Kubernetes supported version in the very first role (core).
  • Users who aren't installing Kubernetes do not need/want this downgrade
• Since, Docker works great on HypriotOS 1.5 without any changes (for non k8s use)
  • I put the docker pinning/downgrade tasks in the kubernetes role
  • This only requires adding two tasks to the beginning of the kubernetes:docker file:
    • copy the pinning template for supported docker version
    • downgrade docker to pinned version
  • These are followed by the four tasks that were already in the file
  • I removed all "overlay related" tasks (from all roles) and have no issues before or after Docker downgrade
  • The only docker task remaining in the core role is "copy docker_profile.sh" in core:system
  • I kept it because it add's value in every scenario
  • It doesn't break anything in any scenario

There may be a few others that I find. If so, i'll post them in a separate response.

I hope this helps, Robert.

[rhuss]

Hey @robertpeteuil,

thanks a lot for your detailed comments. As you probably have noticed, I'm not really an Ansible expert (tbh, these are my first Ansible playbooks ;-).

I will go through them, and adapt the roles accordingly. 'hope that I can make some progress with some additional addons like rook or traefik soon, too (though no promises, as this year seems to be a crazy busy one).

thanks again ...

[robertpeteuil]

I'm happy to help.

I'd really like to find a way to leverage each other's work,

• It appears we both have shared goals - build clusters for demos & tutorials.
• I could benefit if you focused your time on new roles like rook and traefik, instead of re-creating what I've already got working, and vice versa

Let's start by sharing what i've done thus far - I just need to get it cleaned up and posted.

• I'm building it piece by piece, so i'm only up to kubernetes cluster creation
• but, the core section likely contains most of the stuff of interest

If I don't have time to get it cleaned up for public posting (making a readme), I'll upload as private & make you a collaborator.

[robertpeteuil]

One other item I just remembered.

You have a couple of tasks that reboot the nodes then wait for them to come back online. I found it in the "kubernetes-full-reset", but it might be used in other places.

For me it always waited until the timeout period (60 seconds), until I added "port=22" to the command. Now my nodes return after a reboot in 15-20 seconds (they're overclocked U3 cards.).

wait_for: host={{ inventory_hostname }} port=22 state=started delay=10 timeout=60

[rhuss]

Thanks a lot for everything, I took over most of your suggestions unaltered and tested it afresh on a vanilla setup. Works out quite smoothly ;-)

optional: may want to verify the dhcp tasks still work as expected Only mentioned because I've not tested them because I've implemented static-IP assignment differently

Not sure what you mean with this.

---

BTW, I would be very happy if we could add your changes, too and eventually even rename the repo to something more suitable (as currently, openshift is a bit out of scope, as long as I can't get hands on a supported ARM binary). Also, if you are feeling committed to the repo, too, I'm perfectly fine to grant write permissions to the project, too, so that we can work in parallel on it. Any suggestions how to proceed?

[robertpeteuil]

Thanks for your kind words, and acknowledgements in the commits.

My comment about "dhcp tasks" just meant that I had not tested that code with Hyp 1.5 - I just meant "FYI, I haven't checked this"

Thank you for your offer and I definitely like to combine our efforts. Unfortunately, I looked into it and merging isn't possible at the moment as our solutions don't map 1:1 right now. (I explain why below) I think the biggest issue is the differences in how we define our inventories. It creates variable mismatches between our solutions (which beaks templates and other things). For example:your "name" == my "inventory_hostname", and your "inventory_hostname" == my "ansible_host".

In the meantime, here's two plays that can replace a shell script you have. As any good ansible script - it only changes it once, so you don't have to worry about the machine-id changing if the script is re-ran (unlike the shell script). I run this in my equivalent of your core role, in the system file:

- name: System | Get machine-id
  shell: cat /etc/machine-id
  register: node_machine_id
  changed_when: False

- name: System | Change machine-id if default
  shell: 'dbus-uuidgen > /etc/machine-id'
  when: node_machine_id.stdout == "d7866798b7e648d38289364f159bec68"
  register: machine_id
  ignore_errors: True

# Then, I also trigger a reboot later on:
  when: machine_id|changed

As I started to mention above - the reason my playbooks and inventory are different is that I'm designing it from the ground up to support multiple "Clusters" of different types (at the same time). This adds a lot of complexity, and I basically started from scratch. I've been able to leverage from your repo and 2 others, but I still need to write a lot of stuff myself. So, I need to get my solution flushed out a bit - then we can determine what and/or how we can merge.

Wow - this ended up being a long post. Sorry for creating the TL/DR entry.

Cheers, Robert.