Open sherifabdlnaby opened 3 years ago
Appreciate this issue being documented. Just saved sometime by switching that wait configuration.
Spent so much time investigating this and looking into Tiller logs, etc... manual install worked fine, but the hooks never run when deployed from my TF module.
Hi All,
To use helm_release
to deploy airflow to a clean environment just set wait=false
and delopment will work.
resource "helm_release" "airflow" {
repository = "https://airflow-helm.github.io/charts"
chart = "airflow"
name = "airflow"
version = "8.0.9"
namespace = "airflow"
wait = false
}
I don't know what causes this bug, but I can tell what happens on the deployment and airflow side I hope it helps. Manual installation steps:
hooks
and this will trigger the airflow db init
which creates the database schemacheck-db
and wait-for-db-migrations
helm_release installation steps:
hooks
wont run and airflow db init
will not trigger, database schema will not be createdcheck-db
and wait-for-db-migrations
. check-db
will pass however the wait-for-db-migrations
will stuck in a "bootloop" with the following error message.
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 54, in check_migrations
db.check_migrations(timeout=args.migration_wait_timeout)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 598, in check_migrations
raise TimeoutError(f"There are still unapplied migrations after {ticker} seconds.")
TimeoutError: There are still unapplied migrations after 60 seconds.
This is cased by airflow check_migrations
function which does the following check source_heads == db_heads
. The 'db_heads' will be always empty because the database were newer initialised and schema was never created.
face same issue with bitnami/kong too
Bitnami Kong also use post-install
to create a job, but not show in events
kubectl -n <your_namespace> get events --sort-by='{.lastTimestamp}'
and thx @andormarkus workaround solution.
I faced this issue as well and it took a minute to realize that it was my setting the --atomic
flag in the helm command that was causing the problem- because setting --atomic
apparently also sets --wait
internally.
I also was seeing this issue!
I have the same issue with my own charts. If i remove the hook it works fine. otherwise, helm_release just ignore it.
Same problem here on my own chart!
The same problem with 2.8.0 and CloudBees sidecar helm chart:
resource "helm_release" "sidecar" {
chart = "cloudbees-sidecar-injector"
repository = "https://charts.cloudbees.com/public/cloudbees"
version = var.cbci_sidecar_chart_version
name = "cloudbees3-injector"
namespace = var.cbci_sidecar_namespace
create_namespace = true
}
Got to replicate the install of charts that involve post-install hooks and found that with helm CLI that the chart will not install if --wait
is set. Not much can be done in the provider since this is the expected behavior in helm when attempting to install charts that involve hooks, making sure that wait = false
is important.
If anyone is coming here from data-on-eks self-managed-airflow, is testing locally, and already has wait = false
in their airflow_helm_config
, the following steps seemed to work for me (given by the issue submission template):
.terraform
directory: rm -rf .terraform/
terraform init
This allowed me to successfully deploy updates using helm_release
.
This bug also affects the official Mastodon Helm chart.
Yeh, unfortunately the only way to avoid this is:
resource "helm_release" "release-name" {
wait = false
atomic = false
...
}
If you still want to use the wait
feature, airflow has actually documented how to do so:
https://airflow.apache.org/docs/helm-chart/stable/index.html#installing-the-chart-with-argo-cd-flux-rancher-or-terraform
Terraform, Provider, Kubernetes and Helm Versions
Helm chart with post-install hooks will nor be installed succsefully with
wait = true
, the provider behavior waits for installation to be READY, some charts won't get to the READY state without the post install step. eventually timeout is reached. Provider then exists *indicating a succsefully installation and will not install the post-install manfists.Try install https://github.com/apache/airflow/tree/master/chart (you can download the directory instead of installing the entire repository).
You'll notice that the https://github.com/apache/airflow/blob/master/chart/templates/migrate-database-job.yaml is not installed by Helm when used by
helm_release
and usewait = true
. Kindly note theannotations."helm.sh/hook": post-install,post-upgrade
Affected Resource(s)
Terraform Configuration Files
Steps to Reproduce
terraform apply
Expected Behavior
Chart Installed Successfully
Actual Behavior
The chart runs and waits until
timeout
is passed, then it outputs that 1 resource created successfully, but when you check the chart it failed, and checking the Helm release, not all manifests are installed.Important Factoids
None.
Community Note