wellcometrust / reach

Wellcome tool to parse references scraped from policy documents using machine learning
MIT License
25 stars 4 forks source link

Split airflow into a base image / Split airflow and web application #413

Closed jdu closed 3 years ago

jdu commented 4 years ago

In order to simplify our depdency management and to create a better separation of concern in the code-base, we need to build a new airflow base image that will allow us to install our requirements for the DAGs in both Reach and other projects without needing to re-build the entire airflow toolchain each time we need a build, as well as reduce the overall size for the web container image.

The new airflow container image can be set up in a new repo and managed separately from the other repositories. This will allow airflow to be specified as a dependency and makes it less tightly ingrained into the code-base itself.

Overall, this should significantly speed up container builds, reduce issues around dependency versioning and management as well a simplify reasoning about the frontend of Reach as an application itself.

Implementation Details

jdu commented 4 years ago

Would be addressed by #419

jdu commented 3 years ago

No longer relevant as airflow is deprecated.