refactor cbs etl process

[OriHoch]

according to this document (cbs etl section): https://docs.google.com/document/d/1LirLg1u7B3ekvcjetj6LhVExijVqctMZ83fJHUYzlrI/edit#heading=h.rxg36ie7m1yr

also, deploy an Airflow server to run this etl process

[ziv17]

Hi, What is the feature we are implementing, or the issue we are resolving? Without answer to the above question I think it is hard to explain why are we

• Writing mid results to disk. It adds code that needs to be maintained, and slows the process.
• Introducing an additional library and data format to write/read data to disk. Especially as they are many orders of magnitude less popular from the packages we already use for these tasks.

[OriHoch]

The main goal is to improve the ability to make changes and to maintain the code in the long-term

• Writing mid results to disk allows to split the full process to separate, independent tasks which makes it easier to maintain, debug and make changes to. Regarding speed of running the process, why do you think it's important?
• I just use the package I'm used to working with to speed up development, but it plays a very small part in the process, most of the code is standard Python and can be used with any package

[OriHoch]

issue is done, ready for review

Airflow server was deployed (ask @atalyaalon for credentials):

• prod: https://airflow.anyway.co.il/
• dev: https://dev-airflow.anyway.co.il/

ETL data directory is availble at:

• prod: https://airflow-data.anyway.co.il/
• dev: https://dev-airflow-data.anyway.co.il/

Summary of changes:

• anyway-etl repository - contains all ETL tasks and configuration of the Airflow server
• anyway repository changes are in PR #1872 - see PR description for details
• Main Kubernetes cluster changes were done in the following commits:
  • https://github.com/hasadna/hasadna-k8s/commit/dcdd417a618b9ddbc8282c747e2e18050106fe2a
  • https://github.com/hasadna/hasadna-k8s/commit/95f867650bc893421f4c54869bf77ddf68eabee2