search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.63k stars 5.37k forks source link

"cluster add" on rancher cluster fails with "REST config invalid: the server has asked for the client to provide credentials" #1237

Open toolbar23 opened 5 years ago

toolbar23 commented 5 years ago

We want to use ArgoCD with two clusters created using Rancher RKE (running on our own hardware).

ArgoCD is running on Cluster "int01". Deployments within the same cluster work fine.

When we try to add a second cluster "rz01" via the CLI we receive an error:

pm$ argocd cluster add rz01 --server localhost:8080 --insecure --loglevel DEBUG 
INFO[0000] ServiceAccount "argocd-manager" already exists 
INFO[0000] ClusterRole "argocd-manager-role" updated    
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" already exists 
FATA[0001] rpc error: code = Unknown desc = REST config invalid: the server has asked for the client to provide credentials

In our Rancher log files we see 

"2019-03-07T15:41:57.000Z","fluentd-hkn7l","fluentd-hkn7l Rancher: log:10.130.94.100 - [10.130.94.100] - - [07/Mar/2019:15:41:53 +0000] ""GET /k8s/clusters/c-bsjhn/version?timeout=32s HTTP/2.0"" 401 62 ""-"" ""argocd-server/v0.0.0 (linux/amd64)

And the argocd-server has this information regarding the failed request

time="2019-03-07T15:46:51Z" level=info msg="received unary call /cluster.ClusterService/Create" grpc.method=Create grpc.request.claims="{\"iat\":1551972868,\"iss\":\"argocd\",\"nbf\":1551972868,\"sub\":\"admin\"}" grpc.request.content="cluster:<server:\"https://rancher.ariva.k8s/k8s/clusters/c-bsjhn\" name:\"rz01\" config:<username:\"\" password:\"\" bearerToken:\"eyJhbGciOiJSUz......... CUT .......oDOBb1khnGj91E8jWHeHcy2ZTGODIUYJY9H2Z9UBJkxsNGPH3dEiat1wQC6TRFY\\njVfj4bdBBM00gvkqcHSrsY+rL6D9dUahl0eyj2frl8HvdXYKkhG2X6Lnk9Yx2t5J\\n6rKtRG78bD7U0/tnYLtIPrvaaGc/CKcScH6IksT/yAIjWA==\\n-----END CERTIFICATE-----\" > > connectionState:<status:\"\" message:\"\" > > " grpc.service=cluster.ClusterService grpc.start_time="2019-03-07T15:46:51Z" span.kind=server system=grpc
7.3.2019 16:46:51 time="2019-03-07T15:46:51Z" level=error msg="finished unary call with code Unknown" error="REST config invalid: the server has asked for the client to provide credentials" grpc.code=Unknown grpc.method=Create grpc.service=cluster.ClusterService grpc.start_time="2019-03-07T15:46:51Z" grpc.time_ms=14.475 span.kind=server system=grpc
toolbar23 commented 5 years ago

tried 0.12.0-rc3 today with similar results.

first execution gave

pm$ ./argocd cluster add rz01
INFO[0000] ServiceAccount "argocd-manager" already exists 
INFO[0000] ClusterRole "argocd-manager-role" updated    
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" already exists 
FATA[0020] Failed to establish connection to 127.0.0.1:8080: context deadline exceeded

and then always 

pm$ ./argocd cluster add rz01
INFO[0000] ServiceAccount "argocd-manager" already exists 
INFO[0000] ClusterRole "argocd-manager-role" updated    
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" already exists 
FATA[0000] rpc error: code = Unknown desc = REST config invalid: the server has asked for the client to provide credentials
jessesuen commented 5 years ago

It appears that token auth to rancher clusters may be a bit special/different. I believe you may be hitting the following issue: https://github.com/rancher/rancher/issues/14997

Can you read through that issue and see if the discussion applies to your situation?

igaskin commented 5 years ago

I'm also currently working through the same problem, with adding non-rancher clusters. FATA[0000] rpc error: code = Unknown desc = REST config invalid: no such host. I've had success adding clusters in the past, perhaps it is breaking because I am now using microk8s?

igaskin commented 5 years ago

I'm also currently working through the same problem, with adding non-rancher clusters. FATA[0000] rpc error: code = Unknown desc = REST config invalid: no such host. I've had success adding clusters in the past, perhaps it is breaking because I am now using microk8s?

My problem was not with argo, but with coredns. I had to edit the configmap to add forward statements for an internal corporate domain.

papanito commented 5 years ago

Any updates on this one? Facing the same issue.

papanito commented 5 years ago

So when reading trough the referenced issue

... but containing a token Rancher has no knowledge of.. .... ... removes a layer of protection and introduces direct exposure of all clusters to arbitrary requests from anyone that can reach the server container....

We have exactly this problem, we don't want to proxy the requests thus it would be nice if one could provide the account which is used to connect to kubernetes.

papanito commented 5 years ago

@jessesuen do you think argocd could be changed in a way so that it could interact with ranger as well. I had a look at the code but go is new to me - currently this issue is blocking me from further evaluating argocd for our environment :cry:

samuelmak commented 5 years ago

Any updates on this issue? We are facing the same issue.

alexec commented 5 years ago

What could really help this along is someone with access and experience to Rancher to review the issues @jessesuen mentioned and investigate.

LinAnt commented 5 years ago

I am pretty sure this is resolved if one chooses to use the Authorized Cluster endpoint and a generated API key.

I am planning to test ArgoCD in the coming days.

LinAnt commented 5 years ago

I just verified that ArgoCD works correctly when using an Authorized Cluster Endpoint with a Cluster Scoped API key.

samuelmak commented 5 years ago

@LinAnt May I know the detail procedures for how to use authorized cluster endpoint with API key?

LinAnt commented 5 years ago

https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/options/#authorized-cluster-endpoint

samuelmak commented 5 years ago

Confirmed that authorized cluster endpoint can be one of the alternative solution. However, it will be good if Argo CD can directly support Rancher cluster.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

papanito commented 5 years ago

Still interested in a fix for this

amioranza commented 5 years ago

Interested in a real fix, not this alternate strange fix that bypasses Rancher to access directly the cluster.


INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" already exists
FATA[0001] rpc error: code = Unknown desc = REST config invalid: the server has asked for the client to provide credentials```
LinAnt commented 5 years ago

What do you mean with real fix? Authorized cluster endpoints literally is the way Rancher recommends handling these things? I am not following.

amioranza commented 5 years ago

A real fix to me is just run argocd cluster add and argo deal with any other complexity to connect to the cluster. Another fix could be a session on the documentation how to proceed with Rancher. This authorized endpoints is not enable by default, to me is the same to not out-of-box support. Using this way you need to create another load balancer pointing to the directly to kubernetes nodes with the api server role (masters), you will have a spof without a balancer and it need to use the internal kubernetes certificate that is not the same used by the users to connect the cluster via Rancher. You said that is the recommended, I think that is a workaround.

Maybe this documentation session could help Rancher users to go ahead without the need to digging around the solution, but native Rancher support with just the command to add the cluster is what I call real fix.

eroji commented 4 years ago

There is a workaround without having to enable authorized endpoints. You can use this script: https://gist.githubusercontent.com/superseb/f6cd637a7ad556124132ca39961789a4/raw/a833ce5548eded9b110f1b5d4dc1896562338975/get_kubeconfig_custom_cluster_rancher2.sh Get the local admin kubeconfig for the Rancher managed custom cluster, then use ArgoCD CLI to add the cluster using the kubeconfig. You will likely need to edit the cluster secret created after though as it created the cluster with the name "local". You just need to echo name_you_want | base64 and replace the name value in the secret.

janeczku commented 4 years ago

Anyone still looking for a clean solution that doesn't involve doing stuff with the local admin account (no pun intended 😄 ): https://gist.github.com/janeczku/b16154194f7f03f772645303af8e9f80

To summarize:

Suggestion: It would be great if argocd cluster add would support specifying a bearer token that was created out of band, e.g. a service account token or Rancher API token.

travis-sobeck commented 3 years ago

Also per https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/options/#authorized-cluster-endpoint "The authorized cluster endpoint only works on Rancher-launched Kubernetes clusters. In other words, it only works in clusters where Rancher used RKE to provision the cluster. It is not available for clusters in a hosted Kubernetes provider, such as Amazon’s EKS." So that doesn't work with hosted clusters.

ron1 commented 2 years ago

Once Rancher Provisioner issue https://github.com/rancher/rancher/issues/38053 is resolved, an operator such as https://github.com/dntosas/capi2argo-cluster-operator should be able to fully automate this manual operation.

nitishfy commented 1 month ago

can we close this issue now?