airflow 구축

[heehehe]

• [x] dag 파일 생성 (일별로 bigquery 업데이트 되도록)
• [x] airflow 환경 세팅
  • 화림님 k8s 환경 (https://github.com/HWALIMLEE/airflow_tutorial)
  • OR cloud composer 이용 (https://cloud.google.com/composer) https://console.cloud.google.com/marketplace/product/google/composer.googleapis.com?returnUrl=%2Fcomposer%3Freferrer%3Dsearch%26hl%3Dko%26project%3Dwoven-backbone-404105&hl=ko&project=woven-backbone-404105
• [x] gcp vm 테스트 (docker compose 이용)
• 추후 진행: 윤호님 ansible 기반으로 구축해주신 vm 기반으로 세팅 완료시키기 (#39)

[heehehe]

[heehehe]

• gcp vm에서 airflow 세팅해서 실행하기
• References
  • https://yamerong.tistory.com/59
  • (docker compose로 진행) https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#fetching-docker-compose-yaml

[heehehe]

• linux에서 chromedriver - chrome 버전 호환 오류로, geckodriver - firefox 이용
• https://github.com/mozilla/geckodriver/releases