does not delete untracked releases in namespaces that don't have a tiller installed

[despairblue]

When only using on tiller instance (e.g. in kube-system) it's possible to deploy applications to other namespaces than kube-system but they're not removed when they are removed from the DSF. That is because helmsman only check for untracked releases in the namespace where tiller is deployed:

2018/11/01 14:56:48 INFO: checking if any Helmsman managed releases are no longer tracked by your desired state ...
2018/11/01 14:56:48 INFO: getting helm releases which are managed by Helmsman in namespace [[ kube-system ]].
2018/11/01 14:56:49 INFO: getting helm releases which are managed by Helmsman in namespace [[ kube-system ]].
2018/11/01 14:56:49 INFO: no untracked releases found.

Also, awesome work on this, and let me know if I can help with getting this fixed :)

[despairblue]

That happens here. This is iterating over the namespaces making sure they have useTiller or installTiller. So I guess I can fix this by adding useTiller. Let me check.

[despairblue]

Nevermind, useTiller means it's using a tiller in that namespace, but I only want one instance in kube-system.

So that here should not check if useTiller or installTiller is true.

[sami-alajrami]

thanks @despairblue :) The reason why it looks only in namespaces with installTiller or useTiller is because in a multi-tenant cluster, you don't want to delete other people's releases from their namespaces. So the cleanup looks only for namespaces in your desired state file with a Tiller deployed or used.

In your logs snippet, I see that it is searching in kube-system but not finding anything. You can run helmsman with --verbose which will give you a helm command you could use to search for untracked releases. Could you please verify what that command result is?

[sami-alajrami]

Also, to clarify, the tracking is done by labelling the helm release state (in secrets or configmaps) in the namespace where Tiller is deployed. So if you are using Tiller from kube-system , these labeled secrets/configmaps will be in kube-system.

[despairblue]

the command is kubectl get configmap -n kube-system -l MANAGED-BY=HELMSMAN

and the result of that command is:

``` NAME DATA AGE atlas-api.v1 1 2d atlas-api.v2 1 2d atlas-chicken-bot.v1 1 2d atlas-chicken-bot.v2 1 2d atlas-crm.v1 1 2d atlas-mail-service.v1 1 2d atlas-mail-service.v2 1 2d atlas-pipedrive-service.v1 1 2d atlas-pipedrive-service.v2 1 2d atlas-webhooker.v1 1 2d atlas-webhooker.v2 1 2d atlas-website.v1 1 2d atlas-website.v2 1 4h hotfix-intel-api.v1 1 4h hotfix-intel-chicken-bot.v1 1 4h hotfix-intel-crm.v1 1 4h hotfix-intel-mail-service.v1 1 4h hotfix-intel-pipedrive-service.v1 1 4h hotfix-intel-webhooker.v1 1 4h hotfix-intel-website.v1 1 4h keel.v1 1 4d keel.v2 1 2d keel.v3 1 2d master-api.v1 1 4d master-chicken-bot.v1 1 4d master-crm.v1 1 4d master-mail-service.v1 1 4d master-pipedrive-service.v1 1 4d master-webhooker.v1 1 4d master-website.v1 1 4d master-website.v2 1 4h nev-service.v1 1 5d nev-service.v2 1 2d production-crm.v1 1 4d staging-template-api.v1 1 3d staging-template-chicken-bot.v1 1 3d staging-template-crm.v1 1 3d staging-template-mail-service.v1 1 3d staging-template-pipedrive-service.v1 1 3d staging-template-webhooker.v1 1 3d staging-template-website.v1 1 3d traefik-consul.v1 1 5d traefik-production.v1 1 4d wun-1123-api.v1 1 4d wun-1123-chicken-bot.v1 1 4d wun-1123-crm.v1 1 4d wun-1123-mail-service.v1 1 4d wun-1123-pipedrive-service.v1 1 4d wun-1123-webhooker.v1 1 4d wun-1123-website.v1 1 4d wun-1187-api.v1 1 2d wun-1187-api.v2 1 23h wun-1187-api.v3 1 7h wun-1187-chicken-bot.v1 1 2d wun-1187-chicken-bot.v2 1 7h wun-1187-crm.v1 1 2d wun-1187-crm.v2 1 1d wun-1187-mail-service.v1 1 2d wun-1187-mail-service.v2 1 7h wun-1187-pipedrive-service.v1 1 2d wun-1187-pipedrive-service.v2 1 7h wun-1187-webhooker.v1 1 2d wun-1187-webhooker.v2 1 7h wun-1187-website.v1 1 2d wun-1188-api.v1 1 4d wun-1188-chicken-bot.v1 1 5d wun-1188-crm.v1 1 5d wun-1188-mail-service.v1 1 5d wun-1188-pipedrive-service.v1 1 5d wun-1188-webhooker.v1 1 4d wun-1188-website.v1 1 4d wun-1188-website.v2 1 23h wun-1194-api.v1 1 8h wun-1194-chicken-bot.v1 1 8h wun-1194-crm.v1 1 8h wun-1194-mail-service.v1 1 8h wun-1194-pipedrive-service.v1 1 8h wun-1194-webhooker.v1 1 8h wun-1194-website.v1 1 8h wun-1208-api.v1 1 3d wun-1208-website.v1 1 3d wun-1212-api.v1 1 5d wun-1235-api.v1 1 2d wun-1235-chicken-bot.v1 1 2d wun-1235-crm.v1 1 2d wun-1235-mail-service.v1 1 2d wun-1235-pipedrive-service.v1 1 2d wun-1235-webhooker.v1 1 2d wun-1235-website.v1 1 2d wun-1235-website.v2 1 1d ```

[sami-alajrami]

is the release you deleted from your desired state file still showing in that list?

[despairblue]

Yes. It's everything starting with staging-template.

And the DSF is:

``` [metadata] scope = "cluster staging" maintainer = "danny@wunderflats.com" [settings] kubeContext = "[redacted]" [helmRepos] stable = "https://kubernetes-charts.storage.googleapis.com" incubator = "http://storage.googleapis.com/kubernetes-charts-incubator" local = "http://127.0.0.1:8879" [namespaces] atlas = { } default = { } hotfix-intel = { } wun-1187 = { } wun-1188 = { } wun-1194 = { } wun-1208 = { } wun-1235 = { } [namespaces.kube-system] protected = false installTiller = true [namespaces.master] protected = false [namespaces.production] protected = false [apps.atlas-api] name = "atlas-api" description = "Wunderflats API" namespace = "atlas" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/atlas/api/values.yaml" [apps.atlas-chicken-bot] name = "atlas-chicken-bot" description = "Chicken Bot" namespace = "atlas" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/atlas/chicken-bot/values.yaml" [apps.atlas-crm] name = "atlas-crm" description = "Wunderflats CRM" namespace = "atlas" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/atlas/crm/values.yaml" [apps.atlas-mail-service] name = "atlas-mail-service" description = "Mail Service" namespace = "atlas" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/atlas/mail-service/values.yaml" [apps.atlas-pipedrive-service] name = "atlas-pipedrive-service" description = "Pipedrive Service" namespace = "atlas" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/atlas/pipedrive-service/values.yaml" [apps.atlas-webhooker] name = "atlas-webhooker" description = "Provides webhooks for Pipedrive and Prismic" namespace = "atlas" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/atlas/webhooker/values.yaml" [apps.atlas-website] name = "atlas-website" description = "Wunderflats Website" namespace = "atlas" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/atlas/website/values.yaml" [apps.grafana] name = "grafana" description = "Grafana" enabled = false chart = "stable/grafana" version = "1.14.0" purge = true test = false namespace = "default" valuesFile = "apps/default/grafana/values.yaml" [apps.keel] name = "keel" description = "keel.sh" enabled = true chart = "local/keel" version = "0.2.0" purge = true test = false protected = false namespace = "default" valuesFile = "apps/default/keel/values.yaml" [apps.prometheus] name = "prometheus" description = "Prometheus" enabled = false chart = "stable/prometheus" version = "7.0.2" purge = true test = false namespace = "default" valuesFile = "apps/default/prometheus/values.yaml" [apps.traefik-consul] name = "traefik-consul" description = "Traefik Consul" enabled = true chart = "stable/consul" version = "3.3.0" purge = true test = false protected = true namespace = "default" valuesFile = "apps/default/traefik-consul/values.yaml" [apps.hotfix-intel-api] name = "hotfix-intel-api" description = "Wunderflats API" namespace = "hotfix-intel" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/hotfix-intel/api/values.yaml" [apps.hotfix-intel-chicken-bot] name = "hotfix-intel-chicken-bot" description = "Chicken Bot" namespace = "hotfix-intel" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/hotfix-intel/chicken-bot/values.yaml" [apps.hotfix-intel-crm] name = "hotfix-intel-crm" description = "Wunderflats CRM" namespace = "hotfix-intel" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/hotfix-intel/crm/values.yaml" [apps.hotfix-intel-mail-service] name = "hotfix-intel-mail-service" description = "Mail Service" namespace = "hotfix-intel" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/hotfix-intel/mail-service/values.yaml" [apps.hotfix-intel-pipedrive-service] name = "hotfix-intel-pipedrive-service" description = "Pipedrive Service" namespace = "hotfix-intel" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/hotfix-intel/pipedrive-service/values.yaml" [apps.hotfix-intel-webhooker] name = "hotfix-intel-webhooker" description = "Provides webhooks for Pipedrive and Prismic" namespace = "hotfix-intel" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/hotfix-intel/webhooker/values.yaml" [apps.hotfix-intel-website] name = "hotfix-intel-website" description = "Wunderflats Website" namespace = "hotfix-intel" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/hotfix-intel/website/values.yaml" [apps.master-api] name = "master-api" description = "Wunderflats API" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "master" valuesFile = "apps/master/api/values.yaml" [apps.master-chicken-bot] name = "master-chicken-bot" description = "Chicken Bot" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "master" valuesFile = "apps/master/chicken-bot/values.yaml" [apps.master-crm] name = "master-crm" description = "Wunderflats CRM" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "master" valuesFile = "apps/master/crm/values.yaml" [apps.master-mail-service] name = "master-mail-service" description = "Mail Service" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "master" valuesFile = "apps/master/mail-service/values.yaml" [apps.master-pipedrive-service] name = "master-pipedrive-service" description = "Pipedrive Service" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "master" valuesFile = "apps/master/pipedrive-service/values.yaml" [apps.master-webhooker] name = "master-webhooker" description = "Provides webhooks for Pipedrive and Prismic" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "master" valuesFile = "apps/master/webhooker/values.yaml" [apps.master-website] name = "master-website" description = "Wunderflats Website" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "master" valuesFile = "apps/master/website/values.yaml" [apps.production-crm] name = "production-crm" description = "Wunderflats CRM" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "production" valuesFile = "apps/production/crm/values.yaml" [apps.nev-service] name = "nev-service" description = "Wunderflats nev service" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "production" valuesFile = "apps/production/nev-service/values.yaml" [apps.prometheus-operator] name = "prometheus-operator" description = "Manages prometheus and grafana for the whole cluster." enabled = false chart = "stable/prometheus-operator" version = "0.1.7" purge = true test = false namespace = "production" valuesFile = "apps/production/prometheus-operator/values.yaml" [apps.traefik-production] name = "traefik-production" description = "Traefik" enabled = true chart = "local/traefik" version = "1.42.0" purge = true test = false namespace = "production" valuesFile = "apps/production/traefik/values.yaml" [apps.wun-1187-api] name = "wun-1187-api" description = "Wunderflats API" namespace = "wun-1187" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1187/api/values.yaml" [apps.wun-1187-chicken-bot] name = "wun-1187-chicken-bot" description = "Chicken Bot" namespace = "wun-1187" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1187/chicken-bot/values.yaml" [apps.wun-1187-crm] name = "wun-1187-crm" description = "Wunderflats CRM" namespace = "wun-1187" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1187/crm/values.yaml" [apps.wun-1187-mail-service] name = "wun-1187-mail-service" description = "Mail Service" namespace = "wun-1187" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1187/mail-service/values.yaml" [apps.wun-1187-pipedrive-service] name = "wun-1187-pipedrive-service" description = "Pipedrive Service" namespace = "wun-1187" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1187/pipedrive-service/values.yaml" [apps.wun-1187-webhooker] name = "wun-1187-webhooker" description = "Provides webhooks for Pipedrive and Prismic" namespace = "wun-1187" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1187/webhooker/values.yaml" [apps.wun-1187-website] name = "wun-1187-website" description = "Wunderflats Website" namespace = "wun-1187" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1187/website/values.yaml" [apps.wun-1188-api] name = "wun-1188-api" description = "Wunderflats API" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "wun-1188" valuesFile = "apps/wun-1188/api/values.yaml" [apps.wun-1188-chicken-bot] name = "wun-1188-chicken-bot" description = "Chicken Bot" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "wun-1188" valuesFile = "apps/wun-1188/chicken-bot/values.yaml" [apps.wun-1188-crm] name = "wun-1188-crm" description = "Wunderflats CRM" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "wun-1188" valuesFile = "apps/wun-1188/crm/values.yaml" [apps.wun-1188-mail-service] name = "wun-1188-mail-service" description = "Mail Service" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "wun-1188" valuesFile = "apps/wun-1188/mail-service/values.yaml" [apps.wun-1188-pipedrive-service] name = "wun-1188-pipedrive-service" description = "Pipedrive Service" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "wun-1188" valuesFile = "apps/wun-1188/pipedrive-service/values.yaml" [apps.wun-1188-webhooker] name = "wun-1188-webhooker" description = "Provides webhooks for Pipedrive and Prismic" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "wun-1188" valuesFile = "apps/wun-1188/webhooker/values.yaml" [apps.wun-1188-website] name = "wun-1188-website" description = "Wunderflats Website" enabled = true chart = "local/webapp" version = "0.1.7" purge = true test = false namespace = "wun-1188" valuesFile = "apps/wun-1188/website/values.yaml" [apps.wun-1194-api] name = "wun-1194-api" description = "Wunderflats API" namespace = "wun-1194" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1194/api/values.yaml" [apps.wun-1194-chicken-bot] name = "wun-1194-chicken-bot" description = "Chicken Bot" namespace = "wun-1194" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1194/chicken-bot/values.yaml" [apps.wun-1194-crm] name = "wun-1194-crm" description = "Wunderflats CRM" namespace = "wun-1194" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1194/crm/values.yaml" [apps.wun-1194-mail-service] name = "wun-1194-mail-service" description = "Mail Service" namespace = "wun-1194" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1194/mail-service/values.yaml" [apps.wun-1194-pipedrive-service] name = "wun-1194-pipedrive-service" description = "Pipedrive Service" namespace = "wun-1194" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1194/pipedrive-service/values.yaml" [apps.wun-1194-webhooker] name = "wun-1194-webhooker" description = "Provides webhooks for Pipedrive and Prismic" namespace = "wun-1194" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1194/webhooker/values.yaml" [apps.wun-1194-website] name = "wun-1194-website" description = "Wunderflats Website" namespace = "wun-1194" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1194/website/values.yaml" [apps.wun-1208-api] name = "wun-1208-api" description = "Wunderflats API" namespace = "wun-1208" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1208/api/values.yaml" [apps.wun-1208-website] name = "wun-1208-website" description = "Wunderflats Website" namespace = "wun-1208" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1208/website/values.yaml" [apps.wun-1235-api] name = "wun-1235-api" description = "Wunderflats API" namespace = "wun-1235" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1235/api/values.yaml" [apps.wun-1235-chicken-bot] name = "wun-1235-chicken-bot" description = "Chicken Bot" namespace = "wun-1235" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1235/chicken-bot/values.yaml" [apps.wun-1235-crm] name = "wun-1235-crm" description = "Wunderflats CRM" namespace = "wun-1235" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1235/crm/values.yaml" [apps.wun-1235-mail-service] name = "wun-1235-mail-service" description = "Mail Service" namespace = "wun-1235" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1235/mail-service/values.yaml" [apps.wun-1235-pipedrive-service] name = "wun-1235-pipedrive-service" description = "Pipedrive Service" namespace = "wun-1235" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1235/pipedrive-service/values.yaml" [apps.wun-1235-webhooker] name = "wun-1235-webhooker" description = "Provides webhooks for Pipedrive and Prismic" namespace = "wun-1235" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1235/webhooker/values.yaml" [apps.wun-1235-website] name = "wun-1235-website" description = "Wunderflats Website" namespace = "wun-1235" enabled = true chart = "local/webapp" version = "0.1.7" purge = false test = false valuesFile = "apps/wun-1235/website/values.yaml" ```

[despairblue]

I added some logging and getHelmsmanReleases actually only returns the release on the last line of the output of kubectl get configmap -n kube-system -l MANAGED-BY=HELMSMAN

Thus the other releases are not checked.

[despairblue]

Fixed it. PR incoming.

[despairblue]

@sami-alajrami thanks for merging. Do you have an idea when you cut a release again?

[despairblue]

@sami-alajrami @luisdavim Sorry for being to impertinent about this, but environments are piling up on our staging cluster because of this :smile:

If the next release if further away I'll just compile a static version for our custom docker image for helmsman. I just wanted to make sure if the next release is just around the corner so I can safe myself the trouble :)