[etcd] Can't access etcd when installing in non-default namespace with default values

[jakeslee]

I found that Chart installed in specific namespace(no default) still has the problem, thought I passed --user root:$ETCD_ROOT_PASSWORD option.

1. Install via: helm install --name etcd bitnami/etcd --namespace kv
2. Run command as NOTES:

   export POD_NAME=$(kubectl get pods --namespace kv -l "app.kubernetes.io/name=etcd,app.kubernetes.io/instance=etcd" -o jsonpath="{.items[0].metadata.name}")
   export ETCD_ROOT_PASSWORD=$(kubectl get secret --namespace kv etcd -o jsonpath="{.data.etcd-root-password}" | base64 --decode)
   kubectl -n kv exec -it $POD_NAME -- etcdctl --user root:$ETCD_ROOT_PASSWORD put /message Hello

3. Return error message:

   {"level":"warn","ts":"2019-11-16T15:34:48.629Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-848f49f0-4402-468d-9007-754400173a4b/127.0.0.1:2379","attempt":0,"error":"rpc error: code = InvalidArgument desc = etcds
   erver: authentication failed, invalid user ID or password"}
   Error: etcdserver: authentication failed, invalid user ID or password
   command terminated with exit code 2

Originally posted by @jakeslee in https://github.com/bitnami/charts/issues/1454#issuecomment-554648417

[alemorcuq]

Hi, @jakeslee.

I'm not able to reproduce your issue. Here is what I did:

• Install etcd:

helm install --name etcd bitnami/etcd --namespace kv

• Get the pod name and the root password:

export POD_NAME=$(kubectl get pods --namespace kv -l "app.kubernetes.io/name=etcd,app.kubernetes.io/instance=etcd" -o jsonpath="{.items[0].metadata.name}")
export ETCD_ROOT_PASSWORD=$(kubectl get secret --namespace kv etcd -o jsonpath="{.data.etcd-root-password}" | base64 --decode)

• Try to put a message without credentials (fails as expected):

kubectl exec -n kv -it $POD_NAME -- etcdctl put /message Hello
{"level":"warn","ts":"2019-11-18T07:46:41.037Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-7f90cfad-318d-46a4-8fd9-28d08d6eba21/127.0.0.1:2379","attempt":0,"error":"rpc error: code = InvalidArgument desc = etcdserver: user name is empty"}
Error: etcdserver: user name is empty
command terminated with exit code 1

• Try to put a message with credentials:

kubectl exec -n kv -ti $POD_NAME -- etcdctl --user root:$ETCD_ROOT_PASSWORD put /message Hello
OK

• Try to get the message:

kubectl exec -n kv -it $POD_NAME -- etcdctl --user root:$ETCD_ROOT_PASSWORD  get /message
/message
Hello

Is it possible that your namespace had some old data that is causing the authentication to fail?

[jakeslee]

I am sorry for that. I can't reproduce the issue again, too. Maybe the old data caused the problem. Thanks for your attention. I'll close the issue.

[fracasula]

Posting as this might help somebody else.

I'm still getting this but it's a bit of an edge case. When I do helm install ... on a brand new cluster with the defaults on a custom namespace, the first time everything is fine. The problem is when I helm uninstall and the helm install ... again.

To reproduce do:

• helm install
• helm uninstall
• helm install

The issue is that helm uninstall does not delete PVCs (see here).

So if you manually delete the PVC after helm uninstall and then try helm install again it works.

I'm getting this while automating some integration tests so it's not a big deal for me at the moment since I can just purge the PVCs after the uninstall but hopefully this post could save some a headache :sweat_smile:

Also, given that we're talking about integration tests it is super easy for me to replicate this, so if somebody wants to look into it regardless I'd be happy to share my tests.

[nilarte]

I was facing simillar issue for ytt / carvel install of etcd. Deleting PVC was a good solution worked for me!