astronomer / astro-cli

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer
https://www.astronomer.io
Other
361 stars 71 forks source link

Adds Container Runtime Binary Autodetection #1738

Closed schnie closed 1 week ago

schnie commented 3 weeks ago

Description

This PR adds the ability for the CLI to autodetect the container runtime binary to use for the commands that utilize containers. If the already existing configuration (containers.binary) is set in the config.yaml file, we use that, otherwise we search file paths in $PATH for docker, then podman. Others can be added if needed.

Previously all commands that used the container runtime would grab the configuration directly (config.CFG.DockerCommand.GetString()). We've replaced all instances of this with calls to a new function that wraps up this configuration and layers on the auto-detection piece.

This is a small piece of a larger project to simplify getting started with Astro CLI. We're also exploring the idea of bundling podman with our package and using it as our default, rather than Docker Desktop. This is part of that effort, but also just useful on its own.

New functionality is in the new files here. The other files are mostly just adapting to the new method signature.

This effectively makes the highlighted command show below unnecessary when using podman.

Screenshot 2024-10-30 at 3 07 29 PM

🎟 Issue(s)

Related to https://github.com/astronomer/astro/issues/24344

🧪 Functional Testing

Tested on Mac and Windows with Docker Desktop, Orbstack (docker binary), and Podman.

Mac OS

  1. astro dev start with no container runtime installed

    ❯ /Users/schnie/repos/astro-cli/astro dev start
    Error: Failed to find a container runtime. See the Astro CLI prerequisites for more information. https://www.astronomer.io/docs/astro/cli/install-cli
  2. astro dev start after running brew install podman Automatically detects podman binary and uses it.

    ❯ podman ps
    CONTAINER ID  IMAGE                                      COMMAND               CREATED       STATUS                   PORTS                               NAMES
    b6bfe6d579f1  docker.io/library/postgres:12.6            postgres              24 hours ago  Up 9 seconds             127.0.0.1:5432->5432/tcp, 5432/tcp  test-dags_434792-postgres-1
    abfede8c9e6b  localhost/test-dags_434792/airflow:latest  bash -c (airflow ...  24 hours ago  Up 9 seconds                                                 test-dags_434792-scheduler-1
    767acd8d3b6a  localhost/test-dags_434792/airflow:latest  bash -c (airflow ...  24 hours ago  Up 9 seconds                                                 test-dags_434792-triggerer-1
    8e927abccec2  localhost/test-dags_434792/airflow:latest  bash -c if [[ -z ...  24 hours ago  Up 8 seconds (starting)  127.0.0.1:8080->8080/tcp, 8080/tcp  test-dags_434792-webserver-1
  3. astro dev start after additionally running brew install orbstack (docker binary) Both binaries are installed and docker is selected, as it's first in our list.

    ❯ docker ps
    CONTAINER ID   IMAGE                             COMMAND                  CREATED        STATUS                    PORTS                      NAMES
    868eba18c5c9   test-dags_434792/airflow:latest   "tini -- /entrypoint…"   27 hours ago   Up 29 seconds (healthy)   127.0.0.1:8080->8080/tcp   test-dags_434792-webserver-1
    d35e13b7616b   test-dags_434792/airflow:latest   "tini -- /entrypoint…"   27 hours ago   Up 30 seconds                                        test-dags_434792-scheduler-1
    b9f4e14b0a4f   test-dags_434792/airflow:latest   "tini -- /entrypoint…"   27 hours ago   Up 30 seconds                                        test-dags_434792-triggerer-1
    868f8a3ce33b   postgres:12.6                     "docker-entrypoint.s…"   3 weeks ago    Up 31 seconds             127.0.0.1:5432->5432/tcp   test-dags_434792-postgres-1
    ❯ podman ps
    Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
    Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:61555: connect: connection refused
  4. astro dev start after installing both binaries, but manually overriding to podman with the config file

    ❯ /Users/schnie/repos/astro-cli/astro config set container.binary podman -g
    Setting container.binary to podman successfully
    ❯ podman ps
    CONTAINER ID  IMAGE                                      COMMAND               CREATED       STATUS                   PORTS                               NAMES
    b6bfe6d579f1  docker.io/library/postgres:12.6            postgres              24 hours ago  Up 9 seconds             127.0.0.1:5432->5432/tcp, 5432/tcp  test-dags_434792-postgres-1
    abfede8c9e6b  localhost/test-dags_434792/airflow:latest  bash -c (airflow ...  24 hours ago  Up 9 seconds                                                 test-dags_434792-scheduler-1
    767acd8d3b6a  localhost/test-dags_434792/airflow:latest  bash -c (airflow ...  24 hours ago  Up 9 seconds                                                 test-dags_434792-triggerer-1
    8e927abccec2  localhost/test-dags_434792/airflow:latest  bash -c if [[ -z ...  24 hours ago  Up 8 seconds (starting)  127.0.0.1:8080->8080/tcp, 8080/tcp  test-dags_434792-webserver-1

Windows OS

  1. Uninstalled podman and attempted to astro dev start. We see the helpful error message. Screenshot 2024-10-31 at 11 25 35 AM

  2. Installed podman and attempt to astro dev start again. Once podman is installed, we can successfully run astro dev start and view the webserver in the browser. Screenshot 2024-10-31 at 11 27 06 AM Screenshot 2024-10-31 at 11 28 02 AM

📋 Checklist

schnie commented 3 weeks ago

Tagging @lzdanski since this can help simplify the podman docs a bit.

schnie commented 3 weeks ago

@rob-1126 I think you originally add the podman binary support. Any concerns here?

lzdanski commented 1 day ago

@yanmastin-astro Move step 2 to a "tip" box and note that it's required for versions less than 1.31. In release notes, add a section called ### Behavior changes that this step is no longer necessary. In additional improvements this bit: the ability for the CLI to autodetect the container runtime binary to use for the commands that utilize containers.