search

burmilla / os

Tiny Linux distro that runs the entire OS as Docker containers
https://burmillaos.org
Apache License 2.0
210 stars 13 forks source link

How/where you use Rancher OS? Please share your configuration? #6

Closed olljanat closed 6 months ago

olljanat commented 3 years ago

Rancher OS supports quite a lot of different kind of platforms and configurations. I don't want maintain anything which no one is using so please comment to this issue using template below (copy/paste it to comment and answer to questions) and share how you currently using Rancher OS and/or you are planning to use community maintained version if that configuration would be supported.

I will then based on this feedback remove those non-used parts at least from first version and enable them back later if someone request about them later.

**Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.)**

**Which processor architecture you are using?**

**Do you use some extra hardware? (GPU, etc)?**

**Which console you use (default, ubuntu, centos, etc..)**

**Do you use some service(s) which are not enabled by default?**

**Have you installed some extra tools to console?**

**Do you use some other customizations?**

**Please share copy of your cloud-init (remember remove all sensitive data first)**
<replace with sudo ros config export output>

EDIT: Decisions made so far:

olljanat commented 3 years ago

Where are you running/planning to run Rancher OS/community maintained version of it? (docker-machine, AWS, GCE, baremetal, etc.) On Nutanix AHV

Which processor architecture you are using? amd64

Do you use some extra hardware? (GPU, etc)? No

Which console you use (default, ubuntu, centos, etc..) Ubuntu

Do you use some service(s) which are not enabled by default? No

Have you installed some extra tools to console? open-iscsi, iputils-ping

Do you use some other customizations?

Please share copy of your cloud-init (remember remove all sensitive data first)

rancher:
  console: ubuntu
  docker:
    engine: docker-19.03.11
    extra_args:
    - --label
    - type=worker
    - --label
    - node=1
    - --log-opt
    - max-size=5m
    - --log-opt
    - max-file=20
    - --label
    - env=dev
  environment:
    EXTRA_CMDLINE: /init
  hypervisor_service: false
  network:
    post_cmds:
    - iptables -i eth1 -A INPUT -m state --state ESTABLISHED -j ACCEPT
    - iptables -i eth1 -A INPUT -j DROP
  services_include:
    hyperv-vm-tools: false
  state:
    dev: LABEL=RANCHER_STATE
    wait: true
  sysctl:
    fs.inotify.max_user_watches: 1048576
    vm.max_map_count: 262144
runcmd:
- - mount
  - -t
  - ext4
  - LABEL=USER_DOCKER
  - /var/lib/docker
ssh_authorized_keys:
- ssh-ed25519 <key>
write_files:
- content: |+
    #!/bin/bash
    <some commands>
    exit 0
  owner: root
  path: /etc/rc.local
  permissions: "0755"
the-bort-the commented 3 years ago

Where are you running/planning to run Rancher OS/community maintained version of it? (docker-machine, AWS, GCE, baremetal, etc.) I would like to ultimately have Rancher run Docker and Kubernetes on a Raspberry PI Model B 8GB. I have initially tried installing everything on the Official Raspberry Pi OS Full (32-bit) from Raspberry Pi themselves. Which processor architecture you are using? ARM Do you use some extra hardware? (GPU, etc)? I don't believe so; trying to keep everything contained to this one Pi Which console you use (default, ubuntu, centos, etc..) ubuntu Do you use some service(s) which are not enabled by default? Eventually there would be a VPN service installed, Home Assistant, Pi-Hole, stuff like that Have you installed some extra tools to console? nope Do you use some other customizations? I don't believe so. Python, Ansible if those count Please share copy of your cloud-init (remember remove all sensitive data first)

Don't have one yet
ToeiRei commented 3 years ago

Where are you running/planning to run Rancher OS/community maintained version of it? (docker-machine, AWS, GCE, baremetal, etc.) docker vm, raspi

Which processor architecture you are using? amd64, arm64

Do you use some extra hardware? (GPU, etc)? no

Which console you use (default, ubuntu, centos, etc..) debian

Do you use some service(s) which are not enabled by default? no

Have you installed some extra tools to console? git

Do you use some other customizations? no

Please share copy of your cloud-init (remember remove all sensitive data first)

tomaswarynyca commented 3 years ago
PrplHaz4 commented 3 years ago

Where are you running/planning to run Rancher OS/community maintained version of it? (docker-machine, AWS, GCE, baremetal, etc.)

Which processor architecture you are using?

Do you use some extra hardware? (GPU, etc)?

Which console you use (default, ubuntu, centos, etc..)

Do you use some service(s) which are not enabled by default?

Have you installed some extra tools to console?

Do you use some other customizations?

Please share copy of your cloud-init (remember remove all sensitive data first)

hostname: srv-hostname
rancher:
  docker:
    extra_args:
    - -H
    - tcp://0.0.0.0:2375
  environment:
    EXTRA_CMDLINE: /init
  modules:
  - cdc_ether
  - r8152
  - nfs
  - nouveau
  network:
    interfaces:
      mac=50:3e:aa:8a:f8:6f:
        dhcp: true
  services_include:
    kernel-extras: true
    kernel-headers: true
    kernel-headers-system-docker: true
    volume-cifs: true
  state:
    dev: LABEL=RANCHER_STATE
    wait: true
runcmd:
- /home/rancher/restart-usb (script to restart my usb bus after boot - hoping the underlying issue is resolved in later kernels)
- mount -t cifs xxx (bunch of smb mounts here - v2.1 and v3)
ssh_authorized_keys:
- ssh-rsa xxxx
olljanat commented 3 years ago

Special request for @x-jokay @Psayker @jrevillard @etrexel @pierre-emmanuelJ @jianghang8421 @bensallen @drpebcak @mmriis I can see that you have contributed to https://github.com/rancher/os during 2019/2020 and we would like to understand your use cases so we can make sure that those works on BurmillaOS (successor of RancherOS)?

You might be also interested to test our v1.9.0-beta2 version and report findings to #13

jokay commented 3 years ago

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.)

Which processor architecture you are using?

Do you use some extra hardware? (GPU, etc)?

Which console you use (default, ubuntu, centos, etc..)

Do you use some service(s) which are not enabled by default?

Have you installed some extra tools to console?

Do you use some other customizations?

Please share copy of your cloud-init (remember remove all sensitive data first)

hostname: "sample"

rancher:
  docker:
    engine: docker-19.03.11
    tls: true
    registry_mirror: "https://registry-mirror.sample.com"
    insecure_registry: ["registry-mirror.sample.com","registry.sample.com","registry-staging.sample.com"]
  system_docker:
    registry_mirror: "https://registry-mirror.sample.com"
    insecure_registry: ["registry-mirror.sample.com","registry.sample.com","registry-staging.sample.com"]
  services_include:
    qemu-guest-agent: true
    crontab: true
  network:
    dns:
      nameservers:
      - 192.168.1.1
      - 9.9.9.9
      search:
      - sample.com
    interfaces:
      eth0:
        address: 192.168.1.10/24
        gateway: 192.168.1.1
        dhcp: false

ssh_authorized_keys:
  - ssh-rsa ...
jokay commented 3 years ago

And if possible, would be great to have docker-compose as part of the OS available as well 👍🏼

donrudo commented 3 years ago

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.) RPi 4B (4GB and 8GB)

Which processor architecture you are using? arm64

Do you use some extra hardware? (GPU, etc)? PoE HATs

Which console you use (default, ubuntu, centos, etc..) archlinux, centos, alpine(for testing commands directly while debugging Dockerfiles)

Do you use some service(s) which are not enabled by default? micro-HDMI seems to be disabled

Have you installed some extra tools to console? can't get video yet on RPi4B, still trying. USB seems to be not working.

Do you use some other customizations? I usually add /etc/kubernetes and /var/lib/kubelet folders to persistant folders; Also, I usually add to the boot scripts a step to merge: ros config merge -i /user-data

Please share copy of your cloud-init (remember remove all sensitive data first)

#cloud-config
hostname: node-01
burmilla:
  sysctl:
      net.bridge.bridge-nf-call-iptables: 1
      net.ipv4.ip_forward: 1
      net.bridge.bridge-nf-call-ip6tables: 1
  docker:
    tls: false
    extra_args: [ --data-root, /mnt/docker]
    storage_driver: overlay
    log_driver: "json-file"
    log_opts:
      max-size: "100m"
  network:
    dns:
      nameservers:
      - 9.9.9.9
      - 1.1.1.1
    interfaces:
      eth0:
        address: 192.168.1.101/24
        mtu: 1500
        dhcp: false
      wlan0:
        wifi_network: network1
    wifi_networks:
      network1:
        ssid: "Supermeganet"
        psk: "admin/admin"
        scan_ssid: 1
mounts:
  - ['/dev/mmcblk0p3','/mnt/docker','xfs','']
ssh_authorized_keys:
- ssh-rsa ....
EVOTk commented 3 years ago

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.) I'm just discovering, for now on VirtualBox to try, then on Proxmox.

Which processor architecture you are using? amd64 ( N4100 / 8Go RAM )

Do you use some extra hardware? (GPU, etc)? This is not planned, but the CPU that I use has a iGPU

Which console you use (default, ubuntu, centos, etc..) ubuntu

Do you use some service(s) which are not enabled by default? X

Have you installed some extra tools to console? X

Do you use some other customizations? X

Please share copy of your cloud-init (remember remove all sensitive data first)


rancher:
  console: ubuntu
  environment:
    EXTRA_CMDLINE: /init
  network:
    interfaces:
      eth0:
        address: 192.168.1.211/24
        dhcp: false
        gateway: 192.168.1.1
        mtu: 1500
  state:
    dev: LABEL=RANCHER_STATE
    wait: true
ssh_authorized_keys:
- ssh-rsa XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX```
hollec commented 3 years ago

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.) VMware ESXi VM

Which processor architecture you are using? amd64

Do you use some extra hardware? (GPU, etc)? The host server (Intel NUC) has a GPU using the i915 module that I might try using at some point for acceleration.

Which console you use (default, ubuntu, centos, etc..) default, though not tied to this console specifically

Do you use some service(s) which are not enabled by default? kernel-extras open-vm-tools volume-cifs

Have you installed some extra tools to console? docker-compose via these suggestions

Do you use some other customizations? CIFS mounts for containers are enabled via docker volume

Please share copy of your cloud-init (remember remove all sensitive data first)

hostname: rancher
rancher:
  docker:
    engine: docker-19.03.11
  environment:
    EXTRA_CMDLINE: /init
  network:
    dns:
      nameservers:
      - ipv4-dns-ip-here
      search:
      - custom-domain-here
    interfaces:
      eth0:
        address: static-ip/24
        dhcp: false
        gateway: static-gateway-ip
        mtu: 1500
  services_include:
    kernel-extras: true
    open-vm-tools: true
    volume-cifs: true
  state:
    dev: LABEL=RANCHER_STATE
    wait: true
ssh_authorized_keys:
- ...
KebinuChiousu commented 3 years ago

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.)

AWS

Which processor architecture you are using?

amd64

Do you use some extra hardware? (GPU, etc)?

no

Which console you use (default, ubuntu, centos, etc..)

debian

Do you use some service(s) which are not enabled by default?

no

Have you installed some extra tools to console?

docker-compose

Do you use some other customizations?

no

Please share copy of your cloud-init (remember remove all sensitive data first)

EXTRA_CMDLINE: /init
hostname: ip-172-31-57-67.ec2.internal
rancher:
  cloud_init:
    datasources:
    - ec2
  environment:
    EXTRA_CMDLINE: /init
  network:
    dns: {}
  resize_device: /dev/xvda
  services_include:
    xenhvm-vm-tools: true
  state:
    dev: LABEL=RANCHER_STATE
    wait: true
ssh_authorized_keys:
- ...
laghoule commented 3 years ago

We install Kubenetes on RancherOS (iPXE install with ipxe script) via the Rancher RKE tools and Terraform module for Proxmox cluster. Yes we know Kubernetes will drop the Docker engine in 1.22, and we will need to replace it in near future.

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.)

Proxmox

Which processor architecture you are using?

amd64

Do you use some extra hardware? (GPU, etc)?

No

Which console you use (default, ubuntu, centos, etc..)

default

Do you use some service(s) which are not enabled by default?

We load ceph kernel module (rbd) via

Have you installed some extra tools to console?

No

Do you use some other customizations?

Please share copy of your cloud-init (remember remove all sensitive data first)

#cloud-config

# rancher ssh key
ssh_authorized_keys:
  - "my ssh pub key"

# Install RancherOS on /dev/vda (virtio) drive
runcmd:
  - sudo ros install -d /dev/vda -f -c /var/lib/rancher/conf/cloud-config.d/boot.yml

write_files:
  # rc.local script running a boot time
  - path: /etc/rc.local
    permissions: "0755"
    owner: root
    content: |
      #!/bin/sh
      # Load Ceph support
      modprobe rbd

# Rancher configuration
rancher:
  network:
    dns:
      nameservers:
      - x.x.x.x
      - x.x.x.x
    interfaces:
      eth0:
        address: x.x.x.x.24
        gateway: x.x.x.1
        dhcp: false
      eth1:
        address: x.x.x.x/24
        dhcp: false
Raboo commented 3 years ago

I use RancherOS to run Rancher kubernetes cluster via rancher-agent.

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.) KVM/QEMU baremetal

Which processor architecture you are using? amd64

Do you use some extra hardware? (GPU, etc)? No

Which console you use (default, ubuntu, centos, etc..) Barley use the console, so the most lightweight would be preferable. SSH is disabled. I do all config via cloud-init.

Do you use some service(s) which are not enabled by default? qemu-guest-agent

Have you installed some extra tools to console? no

Do you use some other customizations? I install rancher-agent, customize sysctl values and mtu. I PXE boot and use cloud-init to configure ROS/Burmilla(haven't started using burmilla yet)

# Boot a persistent RancherOS to RAM
LABEL rancheros
  KERNEL http://files.example.com/files/os/rancher/vmlinuz rancher.state.dev=LABEL=RANCHER_STATE rancher.state.autoformat=[/dev/vda,/dev/sda] rancher.cloud_init.datasources=['url:https://foreman.example.com/unattended/provision'] vga=0x318 rancher.autologin=tty1 rancher.ssh.daemon=false 
  APPEND initrd=http://files.example.com/files/os/rancher/initrd

Please share copy of your cloud-init (remember remove all sensitive data first)

#cloud-config
hostname: kw-02000cccea29
rancher:
    sysctl:
    vm.overcommit_memory: 1
    vm.panic_on_oom: 0
    kernel.panic: 10
    kernel.panic_on_oops: 1
    kernel.keys.root_maxbytes: 25000000
  network:
    dns:
      nameservers:
        - 10.0.0.11
        - 10.0.0.12
      search:
        - example.com
    interfaces:
      eth*:
        dhcp: true
        # RancherOS defaults to 1450, causes problems for dind(docker in docker) as the dind runs with 1500 MTU.
        mtu: 1500
  docker:
    engine: docker-19.03.15
  services_include:
    qemu-guest-agent: true
  services:
    rancher-agent:
      image: rancher/rancher-agent:v2.5.5
      command: --server https://example.com --token XXX --ca-checksum XXX  --worker
      privileged: true
      net: host
      restart: unless-stopped
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - /etc/kubernetes:/etc/kubernetes
        - /var/run:/var/run
      labels:
        io.rancher.os.after: docker
ssh_authorized_keys:
runcmd:
- wget -O - https://foreman.example.com/unattended/built
h8liu commented 3 years ago

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.) Baremetal, Intel NUC7, Rpi3/4, DigitalOcean Maybe also GCE/AWS in the future. We are also looking into supporting Intel NUC10, where rancheros's vanilla OS install image does not support yet (ethernet not working).

Which processor architecture you are using? amd64, arm64

Do you use some extra hardware? (GPU, etc)? No, not really.

Which console you use (default, ubuntu, centos, etc..) default. console not used that much.

Do you use some service(s) which are not enabled by default? No, not now.

Have you installed some extra tools to console? No.

Do you use some other customizations? We have a custom bootloader hooking to make it boot from UEFI, which rancher OS did not support. We can try to bring UEFI booting (for install from USB) upstream if possible.

Please share copy of your cloud-init (remember remove all sensitive data first)

rancher:
  environment:
    EXTRA_CMDLINE: /init
  state:
    dev: LABEL=RANCHER_STATE
    wait: true
ssh_authorized_keys:
- ... (ssh public keys)
write_files:
- ... (init bash script files, used just docker, wait-for-docker and basic/essential linux command line utils)
matthewkrupnik commented 3 years ago

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.) ESXi and baremetal

Which processor architecture you are using? amd64

Do you use some extra hardware? (GPU, etc)? Couldn't get nvidia GPU working properly so no, but would like to use the nvidia GPU

Which console you use (default, ubuntu, centos, etc..) default

Do you use some service(s) which are not enabled by default?

Have you installed some extra tools to console? No

Do you use some other customizations? I'll include it here, even though it's very much me playing around with stuff... Right now I have a custom compose with glusterfs-server and glusterfs-client services that I install manually after deployment. Once I am happy with it, it'll just go directly into cloud-config as two services.

Please share copy of your cloud-init (remember remove all sensitive data first)

hostname: <hostname>
mounts:
- - <mount address>
  - <mount location>
  - cifs
  - <mount options>,_netdev
- - <mount address>
  - <mount location>
  - nfs4
  - _netdev
rancher:
  environment:
    EXTRA_CMDLINE: /init
  network:
    dns:
      nameservers:
      - <nameserver ip>
    interfaces:
      eth0:
        dhcp: true
  services_include:
    /var/lib/rancher/conf/docker-compose.yml: true
    open-vm-tools: true
    volume-cifs: true
    volume-nfs: true
  state:
    dev: LABEL=RANCHER_STATE
    wait: true
ssh_authorized_keys:
- <my ssh key>
wonleing commented 3 years ago

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.) baremetal， KVM， IOT devices

Which processor architecture you are using? amd64, arm64, mips64el, sw64, loongarch64

Do you use some extra hardware? (GPU, etc)? Not as so far

Which console you use (default, ubuntu, centos, etc..) UOS, deepin linux (https://www.deepin.org/en/)

Do you use some service(s) which are not enabled by default? probably yes. not sure yet.

Have you installed some extra tools to console? not yet

Do you use some other customizations? custom kernel, custom rootfs.

Please share copy of your cloud-init (remember remove all sensitive data first)

no custom need for this part yet.
prologic commented 2 years ago

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.)

I now run Burmilla OS on all my VM nodes, It was RancherOS. This runs on a 3x node PRoxmox VE Hypervisor cluster.

Which processor architecture you are using?

AMD64

Do you use some extra hardware? (GPU, etc)?

No.

Which console you use (default, ubuntu, centos, etc..)

default, but I barely use the console as it is. I typically use Burmilla OS as nodes in a Docker Swarm Clusters.

Do you use some service(s) which are not enabled by default?

No. The open-vm-tools is the only service I rely on.

Have you installed some extra tools to console?

No.

Do you use some other customizations?

No. Only cloud-inint configuration.

Please share copy of your cloud-init (remember remove all sensitive data first)

```#!yaml #cloud-config hostname: xxx ssh_authorized_keys: - ssh-rs a ... rancher: network: dns: nameservers: - 8.8.8.8 - 8.8.4.4 interfaces: eth0: dhcp: false address: a.b.c.d/xx gateway: a.b.c.d mtu: 1500 eth1: dhcp: false ```
ArgonV commented 1 year ago

Hello all, I am proposing the VMware autoformat media be placed back into this distro. The reason being is that in my environment, I bootstrap clusters of ROS nodes with persistent storage, via a node template, using Rancher Server.

Where are you running/planning to run Rancher OS/Burmilla OS? (docker-machine, AWS, GCE, baremetal, etc.)

VMware

Which processor architecture you are using?

Intel Xeon

Do you use some extra hardware? (GPU, etc)?

Attached vmdk disks

Which console you use (default, ubuntu, centos, etc..)

Default

Do you use some service(s) which are not enabled by default?

No

Have you installed some extra tools to console?

VMware Tools that's included

Do you use some other customizations?

Auto-formatting, auto-mounting of vmdk OS disk

Please share copy of your cloud-init (remember remove all sensitive data first)

cloud-init