astronomer / astro-cli

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer
https://www.astronomer.io
Other
351 stars 70 forks source link

Variables/Connections/Pools specified in `airflow_settings.yaml` can't be imported or exported with Podman #1724

Open joaomsa opened 1 month ago

joaomsa commented 1 month ago

Describe the bug Values in airflow_settings.yaml are not respected when using podman as the container binary

When trying to use the astro dev object import or astro dev object export commands while running with podman the commands fail without displaying any error messages.

For the first command, although astro indicates that variables have been successfully imported, the airflow command was never actually run. For the latter we have a fatal error.

These astro commands do not respect the container.binary: podman option specified in the .astro/config.yaml file and instead hardcode the docker executable.

https://github.com/astronomer/astro-cli/blob/main/docker/docker.go#L21

which podman

/opt/podman/bin/podman

which docker

docker not found

cat .astro/config.yaml

project:
name: test-bug
container:
binary: podman

cat airflow_settings.yaml

airflow:
variables:
- variable_name: hello
variable_value: world

astro dev object import


Added Variable: hello

Airflow objects created from settings file


> `astro dev object export`

panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]: github.com/astronomer/astro-cli/settings.ExportPools({0x140001159c0?, 0x15?}) github.com/astronomer/astro-cli/settings/settings.go:571 +0x748 github.com/astronomer/astro-cli/settings.Export({0x140001159c0, 0x40}, {0x102f1869a?, 0x0?}, 0x2, 0x1, 0x1, 0x1) github.com/astronomer/astro-cli/settings/settings.go:435 +0x7c github.com/astronomer/astro-cli/airflow.(DockerCompose).ExportSettings(0x14000262000, {0x102f1869a, 0x15}, {0x102f014ad, 0x4}, 0x0?, 0x0?, 0x0?, 0x0) github.com/astronomer/astro-cli/airflow/docker.go:1156 +0x150 github.com/astronomer/astro-cli/cmd.airflowSettingsExport(0x14000550f08, {0x103f0ed60?, 0x4?, 0x102f015fd?}) github.com/astronomer/astro-cli/cmd/airflow.go:930 +0x108 github.com/spf13/cobra.(Command).execute(0x14000550f08, {0x103f0ed60, 0x0, 0x0}) github.com/spf13/cobra@v1.7.0/command.go:940 +0x648 github.com/spf13/cobra.(Command).ExecuteC(0x140001e6908) github.com/spf13/cobra@v1.7.0/command.go:1068 +0x320 github.com/spf13/cobra.(Command).Execute(0x1034f2160?) github.com/spf13/cobra@v1.7.0/command.go:992 +0x1c main.main() github.com/astronomer/astro-cli/main.go:21 +0x38


> `astro dev run variables export -`

Running: airflow variables export - [2024-09-26T09:43:56.575+0000] {plugin.py:32} WARNING - Astro managed secrets backend is disabled {}



**What CLI Version did you experience this bug?**

> `astro version`

``
Astro CLI Version: 1.29.0
``

**This CLI bug is related to which Astronomer Platform?**

- [x] Astro
- [ ] Software
- [ ] None/Unknown

**What Operating System is the above CLI installed on?**

**🪜 Steps To Reproduce**

Attempt to specify variables/connections/pools in `airflow_settings.yaml` without the docker client installed (using an alternative binary like podman)

**📸 Screenshots**

<!--
If applicable, add screenshots to help explain your problem.
-->
neel-astro commented 4 days ago

Triage: This is a bug in terms of not using the container.binary config when running astro dev object commands. Need to update this logic: https://github.com/astronomer/astro-cli/blob/main/docker/docker.go#L21