airflow infrastructure

[getElementsByName]

endpoint link

• ad1
  • airflow web
  • flower
• cr1
  • airflow web
  • flower

how to apply dag files

• git sync - pdp/develop

deep dive

base

• based on official helm chart

concepts

• https://airflow.apache.org/docs/stable/concepts.html

architecture

• https://airflow.apache.org/docs/stable/executor/celery.html
• https://eng.lyft.com/running-apache-airflow-at-lyft-6e53bb8fccff

ncc

• docker image security
  • base
• service port name
• nubes
  • debugging

pdp

• hadoop client
  • hive spark hadoop
  • xml config
• python version
• helm values
  • gitSync
  • requirements.txt
  • dags path
  • smtp
  • timezone
  • rbac
  • postgresql, redis connection
  • autoscaler

monitoring

• grafana

local container

• docker image layer
• airflow standalone
  • https://airflow.apache.org/docs/stable/start.html
• volume mount
  • dags
  • airflow.cgf
  • airflow.db (sqllite)

TODO

• logrotate
• ncc pipeline