unable to delete helm ceph namespace

[ksingh7]

Is this a request for help?: yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:

[root@admin-node ~]# helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

[root@admin-node ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
[root@admin-node ~]#

Which chart: Ceph

What happened: Unable to release ceph namespace from Helm after a few objects missing in helm status

[root@admin-node ~]# helm list
NAME    REVISION        UPDATED                         STATUS          CHART           NAMESPACE
ceph    1               Sat Jul  7 02:46:40 2018        DEPLOYED        ceph-0.1.0      ceph
[root@admin-node ~]#

[root@admin-node ~]# helm status ceph
LAST DEPLOYED: Sat Jul  7 02:46:40 2018
NAMESPACE: ceph
STATUS: DEPLOYED

RESOURCES:
==> v1/StorageClass
NAME      PROVISIONER   AGE
ceph-rbd  ceph.com/rbd  5d

==> MISSING
KIND           NAME
secrets        ceph-keystone-user-rgw
configmaps     ceph-bin-clients
configmaps     ceph-bin
configmaps     ceph-etc
configmaps     ceph-templates
services       ceph-mon
services       ceph-rgw
daemonsets     ceph-mon
daemonsets     ceph-osd-dev-vdc
deployments    ceph-mds
deployments    ceph-mgr
deployments    ceph-mon-check
deployments    ceph-rbd-provisioner
deployments    ceph-rgw
jobs           ceph-rgw-keyring-generator
jobs           ceph-osd-keyring-generator
jobs           ceph-mgr-keyring-generator
jobs           ceph-mon-keyring-generator
jobs           ceph-mds-keyring-generator
jobs           ceph-namespace-client-key-generator
jobs           ceph-storage-keys-generator

[root@admin-node ~]# helm delete ceph --purge --debug
[debug] Created tunnel using local port: '42271'

[debug] SERVER: "127.0.0.1:42271"

Error: namespaces "ceph" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "ceph"
[root@admin-node ~]#

There are no resources at kubernetes level. I think they got deleted with i first ran helm delete ceph

[root@admin-node ~]# kubectl get all --namespace ceph
No resources found.
[root@admin-node ~]#

Also tried removing helm tiller and doing re-init

kubectl delete deployment -n=kube-system tiller-deploy
helm init --upgrade
kubectl get po -n kube-system
helm delete ceph --purge --debug

What you expected to happen: helm delete --purge should delete the helm namespace

How to reproduce it (as minimally and precisely as possible):

• Deploy ceph using helm chart
• helm delete ceph
• helm delete ceph --purge

Anything else we need to know:

[rootfs]

have you used the rbac rules here

[ksingh7]

@rootfs yes i have used those rules but no effect.

I think i managed to get a workaround for this. Will post here if the testing goes well.

[ksingh7]

• This was a helm issue, basically one needs to find out if there are undeleted resources

  helm delete --purge ceph1 --debug

• Delete those resources manually

  kubectl delete jobs -n ceph ceph-namespace-client-key-cleaner-hw95v

• Open two terminals delete helm application on one helm delete --purge ceph1 --debug while it is executing open the second terminal, list undeleted resources kubectl get all -n ceph, if there appears any undeleted resource, delete it manuallly kubectl delete jobs -n ceph ceph-namespace-client-key-cleaner-hw95v

These instructions worked form me

[root@admin-node tmp]# helm delete --purge ceph --debug
[debug] Created tunnel using local port: '44392'

[debug] SERVER: "127.0.0.1:44392"

release "ceph" deleted
[root@admin-node tmp]#

[root@admin-node tmp]# helm list
[root@admin-node tmp]# echo $?
0
[root@admin-node tmp]#