Client-side Docker image

[blackvvine]

Running weather-mv pipelines is difficult because the dependencies are hard to set up. A client-side Docker image for submitting the jobs can make life easier.

[alxmrs]

Instead of docker, I'd prefer if we implemented #179.

[blackvvine]

I see your point. One of the benefits to providing a Docker image is that it can include gcloud CLI and any necessary scripts in the future as well, so the barrier to entry for the users would be lower. This also simplifies the new README instructions.

The image can look like this:

Dockerfile

FROM ubuntu:22.04
FROM continuumio/miniconda3:4.12.0

# Install latest gcloud cli
RUN apt-get update
RUN apt-get install -y apt-transport-https ca-certificates gnupg curl
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
RUN apt-get update && apt-get -y install google-cloud-cli

# Create Conda environment
WORKDIR /weather
ADD tinyenv.yml /weather/environment.yml
RUN conda env update --name base --file environment.yml --prune

# Install weather-tools
ADD . /weather
RUN pip install --upgrade pip setuptools
RUN pip install .
RUN pip install weather_dl/
RUN pip install weather_mv/
RUN pip install weather_sp/

# make gcloud configurations persist
VOLUME ["./config"]

tinyenv.yml

name: weather-tools
channels:
- conda-forge
dependencies:
- python=3.8
- gdal
- metview-batch

The build takes 5-10 minutes and makes the quickstart easier.

[alxmrs]

I like that as an add on feature for the future. However, the priority is a conda environment because:

• Our north star is for this project to be cloud agnostic instead of wed to GCP
• Docker is a heavy dependency for our target users, who are scientists in the Python ecosystem (over data engineers). For these users, conda is the preferred way to manage environments.