ArgoCD cli login does not work neither for Ingress or LoadBalancer

[ghost]

I can login and use ArgoCD UI via both Ingress and LoadBalancerIP. However, if I try to login via cli it's not working either way:

argocd login --insecure --grpc-web --username admin --password xxxxxxxxxxxx 10.25.0.12
WARNING: server is not configured with TLS. Proceed (y/n)? y
FATA[0031] rpc error: code = Unknown desc = Post "https://10.25.0.12:443/session.SessionService/Create": net/http: TLS handshake timeout
---
argocd login --insecure --grpc-web --username admin --password xxxxxxxxxxx my.domain.com
WARNING: server is not configured with TLS. Proceed (y/n)? y
FATA[0020] rpc error: code = Unknown desc = POST http://my.domain.com:443/session.SessionService/Create failed with status code 400

Ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-dev
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    # nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    # If you encounter a redirect loop or are getting a 307 response code
    # then you need to force the nginx ingress to connect to the backend using HTTPS.
    #
    # nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: my.domain.com
    http:
      paths:
      - backend:
          serviceName: argocd-server
          servicePort: https
        path: /
  tls:
  - hosts:
    - my.domain.com
    secretName: argocd-secret

P.S. ssl-passthrough is set to nginx-ingress-controller in helm via

extraArgs:
  enable-ssl-passthrough: 'true'

LB config:

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
  name: argocd-server
  namespace: argocd
spec:
  clusterIP: 10.99.251.76
  clusterIPs:
  - 10.99.251.76
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 30872
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 31691
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 10.25.0.12

[seangallavan]

I'm having the same problem with port forwarding

[andrei-dascalu]

@Born2Bake is your argocd server running with --insecure flag? I'm not sure if what you're doing make sense .... asking for an insecure connection on an ingress that works on HTTP with HTTPS passthrough to an https backend ?

[ghost]

@Born2Bake is your argocd server running with --insecure flag? I'm not sure if what you're doing make sense .... asking for an insecure connection on an ingress that works on HTTP with HTTPS passthrough to an https backend ?

Hi, ArgoCD was deployed via

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.0.1/manifests/ha/install.yaml

 k get po -n argocd
NAME                                       READY   STATUS    RESTARTS   AGE
argocd-application-controller-0            1/1     Running   1          46h
argocd-dex-server-5dd657bd9-2ldnf          1/1     Running   1          46h
argocd-redis-ha-haproxy-5886787f8b-9bdhs   1/1     Running   1          46h
argocd-redis-ha-haproxy-5886787f8b-9qvlh   1/1     Running   1          47h
argocd-redis-ha-haproxy-5886787f8b-swgsp   1/1     Running   1          47h
argocd-redis-ha-server-0                   2/2     Running   2          46h
argocd-redis-ha-server-1                   2/2     Running   2          47h
argocd-redis-ha-server-2                   2/2     Running   2          47h
argocd-repo-server-f6ccdd667-ddqtz         1/1     Running   1          46h
argocd-repo-server-f6ccdd667-vlkls         1/1     Running   1          47h
argocd-server-56659b9dd8-qvfgc             1/1     Running   1          47h
argocd-server-56659b9dd8-vbt82             1/1     Running   1          46h

As I shared configuration for both LB and Ingress objects, I am not sure what exactly I did wrong with both cause I followed official docs. I tried the same thing without --insecure flag and got the same results:

bn@DESKTOP-IQ0K33P:~$ argocd login --grpc-web --username admin --password xxxxxxxxxxxxxxxx 10.25.0.12
WARNING: server is not configured with TLS. Proceed (y/n)? y
FATA[0032] rpc error: code = Unknown desc = Post "https://10.25.0.12:443/session.SessionService/Create": net/http: TLS handshake timeout
bn@DESKTOP-IQ0K33P:~$ k get ingress -n argocd
NAME                    CLASS    HOSTS                ADDRESS                                  PORTS     AGE
argocd-server-ingress   <none>   deploy.domain.com  172.16.1.135,172.16.1.143,172.16.1.150   80, 443   11d
bn@DESKTOP-IQ0K33P:~$ argocd login --grpc-web --username admin --password xxxxxxxxxxxxxxxxxx deploy.domain.com 
WARNING: server is not configured with TLS. Proceed (y/n)? y
FATA[0023] rpc error: code = Unknown desc = POST http://deploy.domain.com:443/session.SessionService/Create failed with status code 400

[andrei-dascalu]

@Born2Bake Yeah, I'm also not an expert, just brainstorming against what I've experimented with.

Are you running your ingress with passthrough enabled ? (for nginx ingress "--enable-ssl-passthrough" in the args list)?

[ghost]

@Born2Bake Yeah, I'm also not an expert, just brainstorming against what I've experimented with.

Are you running your ingress with passthrough enabled ? (for nginx ingress "--enable-ssl-passthrough" in the args list)?

Yup

image:
  registry: docker.io
  repository: bitnami/nginx-ingress-controller
  tag: 0.44.0-debian-10-r28
extraArgs:
  enable-ssl-passthrough: 'true'

[esn89]

Is it normal to login with the url, then have the cli add the :443 at the end? I am experiencing something similar:
https://github.com/argoproj/argo-cd/issues/14003

[matodrobec]

Hi, I have same problem

 argocd login argo.test.at --username admin --insecure --grpc-web-root-path /argocd/

WARNING: server certificate had error: x509: certificate is valid for ingress.local, not argo.test.at Proceed insecurely (y/n)? y
Password:
FATA[0002] rpc error: code = Unknown desc = POST https://argo.test.at:443/argocd/session.SessionService/Create failed with status code 404

Ingress controller:

  containers:
  - args:
    - /nginx-ingress-controller
    - --enable-ssl-passthrough  

[pachr]

Is it normal to login with the url, then have the cli add the :443 at the end? I am experiencing something similar: #14003

Hello, in my case I am using ArgoCD with GKE.

My service is configured as a ClusterIP with an ingress on top of it and a Load Balancer.

What I can see in my ingress logs when I try to log in like this:

argocd login --username admin --password ******** my.argo.url.org

CLI logs:

FATA[0001] rpc error: code = Unknown desc = POST https://my.argo.url.org:443/session.SessionService/Create failed with status code 404

Ingress logs:

matched_url_path_rule: "UNMATCHED" backend_target_type: "BACKEND_SERVICE" backend_target_name: "k8s1-272033bd-kube-system-default-http-backend-80-c75973eb"

The interesting thing here, is the UNMATCHED value

If I try to curl the URL but without :443 (https://my.argo.url.org/session.SessionService/Create), obviously it fails BUT in the ingress logs I have this instead:

backend_type: "NETWORK_ENDPOINT_GROUP" backend_target_name: "k8s1-272033bd-argocd-argocd-server-443-a367113b" matched_url_path_rule: "/"

Check the matched_url_path_rule, it matches the host pattern from my GCP LoadBalancer and it forwards the request to the correct path and backend (which is ArgoCD here and not the default k8s backend).

So my thought is: yes there's a problem with the CLI which is adding :443 at the end of the ArgoCD hostname... The Load Balancer does not match the defined pattern, and the request goes to the default k8s backend instead of ArgoCD.

I am using the same configuration mentionned in #14003 , ArgoCD deployed on top of GCP, GKE, Load Balancer...

The other thing is the exact same configuration is deployed on top of AWS, and it works... Maybe a problem with ArgoCD CLI and GCP LoadBalancer + Ingress

[pachr]

I tried also to log into the API, and it works, I got the Bearer token.

curl https://my.argo.url.org/api/v1/session -d $'{"username":"admin","password":"**********"}'

So I guess that my ArgoCD instance is ok.

[akoken]

I'm experiencing the same problem with both port-forwarding and ingress. How am I supposed to create a new user or add a cluster without being able to connect using the ArgoCD CLI?

Edit: I gave it another shot. Uninstalling and reinstalling partially fixed my problem. Now I can connnect with port-forwarding. Still I couldn't reach the server via ingress, though. Most likely TLS termination issue on my end.

[bert-jan]

Ran recently also into this issue with argocd v2.8.5 installed via helm. A possible workaround could be switching from ClusterIP to nodePort while keeping the ingress configuration in-place. Connecting to the nodeport port via argocd cli works direct to the worker node. I still have to try it via our external loadbalancer but i expect it to work.

[MarcoPereira7]

I had the same problem using ArgoCD version 2.8.10 on a GKE cluster , being exposed not via ingress, but via google L7 internal load balancer. I added the following rule to my LoadBalancer:

I think the issue is that, when you use argocd cli, it inserts :443 in the request, and if the GCP LoadBalancer does not have that rule containting :443, for some reason, it does not match with any rule and returns a 404. When applying the rule in the image above, i could finally login using the argocd cli.

[xxxx@yyy ~]$ argocd login your-argogrpc.pt --grpc-web
Username: admin
Password:
'admin:login' logged in successfully
Context 'your-argogrpc.pt' updated

Hope it helps.