Build pipeline on AirFlow + Ansible

[abitrolly]

• [ ] Setup 10 Ubuntu 18.04 machines with Ansible and Airflow

• 1 controller

• 9 pilots

• [ ] Access UI on controller

• [ ] Create test job on controller

• [ ] Run test job on 9 pilots and collect result

• [ ] Randomly reboot pilots to check job recovery

• [ ] Modify jobs to fetch heavyweight data file from storage

• [ ] Share new job to running cluster (should it stop old ones?)

[abitrolly]

https://gitlab.com/abitrolly/airflow-net

+1

[abitrolly]

Installed Airflow through Ansible without GPL dependencies.

+3

[abitrolly]

+2 systemd template to autostart airflow webserver on reboot.

LXD port forwarding, finding a way to pass files between Airflow tasks.

[abitrolly]

+2 reading airflow docs and experiments.

[abitrolly]

Ideas for the future. Calculating the time saved. Implement an easy way to register the time that is saved by automation, as a dividend of added benefits by automation, which can be used to fix or improve the automation.

[abitrolly]

+2 PostgreSQL

[abitrolly]

+2 (failed) troubleshooting Scaleway sudo root is not in the sudoers file after bootstrap with Ansible +1 checkout and generate .env +2 (failed) Ansible port forwarding (will upload private SSH keys instead)

[abitrolly]

+1 sudoers fixed with mv /etc/sudoers.dpkg-dist /etc/sudoers +1 git clone with access key and host trust

[abitrolly]

+1 RabbitMQ probe

#!/usr/bin/env python
import pika

USERPASS = "guest guest"

creds = pika.PlainCredentials(*USERPASS.split())
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost',
        credentials=creds))
channel = connection.channel()

[abitrolly]

+1 move DB to different host

[abitrolly]

+1 move RabbitMQ and Redis to a separate host