Using Argo-cd CLI to login returns error

[z-mani]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Using ArgoCD CLI Login command to access the agent in the cluster returns a port-forward error. This issue has only started happening after updating our EKS version from 1.23 to 1.24. Both the runner that is executing these commands and the worker pods are using EKS 1.24 version.

To Reproduce

Commands that are being used are below:

kubectl -n argocd port-forward service/argo-cd-argocd-server 9090:80 & argocd login localhost:9090 --username admin --password $argo_password

Expected behavior

Forwarding from 127.0.0.1:9090 -> 8080 Handling connection for 9090 Handling connection for 9090 WARNING: server is not configured with TLS. Proceed (y/n)? Handling connection for 9090 Handling connection for 9090 Handling connection for 9090 'admin:login' logged in successfully Context 'localhost:9090' updated

Screenshots

Forwarding from 127.0.0.1:9090 -> 8080 Handling connection for 9090 E1220 01:31:38.789021 108 portforward.go:406] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod xxx, uid : failed to execute portforward in network namespace "/var/run/netns/cni-x-x-x-x-x": read tcp4 127.0.0.1:45740->127.0.0.1:8080: read: connection reset by peer E1220 01:31:38.789589 108 portforward.go:234] lost connection to pod Handling connection for 9090 E1220 01:31:38.790969 108 portforward.go:346] error creating error stream for port 9090 -> 8080: EOF time="2022-12-20T01:31:39Z" level=fatal msg="dial tcp 127.0.0.1:9090: connect: connection refused"

Version

Tried using v2.5.5 and 2.4.18

Logs

Paste any relevant application logs here.

[RafalSkolasinski]

I am observing same behaviour and this is quite frustrating

[NadgobKhan]

(almost) same issue here, possibly related to #8613

[JanCizmar]

It looks a workaround is to use kubectl < 1.23. So I did this:

Run bitnami/kubectl:1.22.16 docker image, and forwarding the ports to the docker container and outside of it

docker run --rm --name kubectl -p8089:8089 \
-v ~/.kube/config:/.kube/config bitnami/kubectl:1.22.16 \
port-forward svc/argo-server --address=0.0.0.0 -n argo 8089:443

Now I am able to login with:

argocd login localhost:8089

It's far from ideal, but after spending 2 hours on this I am happy it works.

[RafalSkolasinski]

@JanCizmar so this is related to the way kubectl is doing port-forwarding? o_O

[JanCizmar]

I have no idea, what happens there, but for some reason connecting to Argo API server breaks the port forwarding in later kuebectl versions, which is kinda weird.

[alexallah]

I had encountered a similar issue. The argocd cli doesn't work well with kubectl port-forward.

It turns out that argocd supports native port-forwarding, without the need to use kubectl or anything else. Try adding to every command --port-forward --port-forward-namespace=argocd

Example argocd --port-forward --port-forward-namespace=argocd login --username=admin --password=MY_PASSWORD

This solution worked perfectly in our case, and we completely removed kubectl port-forward

[andrei-dascalu]

@alexallah This workaround didn't work for me:

argocd --port-forward --port-forward-namespace=argocd login --username=admin --password=whatever
E0323 11:52:01.640455   17502 portforward.go:406] an error occurred forwarding 49710 -> 8080: error forwarding port 8080 to pod aed3fdd55f630f74b1a18c139efc08bc0aa4c6af71647a8b9069e5b277cbf646, uid : failed to execute portforward in network namespace "/var/run/netns/cni-694331e8-ff00-f1fa-3875-60c0c321bbf2": read tcp4 127.0.0.1:43858->127.0.0.1:8080: read: connection reset by peer
E0323 11:52:01.641303   17502 portforward.go:234] lost connection to pod
FATA[0001] Failed to establish connection to 127.0.0.1:49710: dial tcp 127.0.0.1:49710: connect: connection refused 

I did manage to use it only once, doesn't happen all the time (but still like 90% of the time)

[welderpb]

@andrei-dascalu same issue for me.. Did you manage to connect to argocd API via port-forwarding?

[prima101112]

same issue here. is there any work around this? neither kubectl port forward and argocd native port forwarding is working

[welderpb]

@prima101112 for me it is working with args: --port-forward --port-forward-namespace argocd --grpc-web --plaintext

[bebosudo]

If you're following the docs and are trying to port-forward remote port 443 onto local 8080, and getting the following error:

$ kubectl -n argocd port-forward svc/argocd-server 8080:443
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080    <--- all good up to now, then we use curl to contact localhost

E1211 10:05:18.904999 1425028 portforward.go:409] an error occurred forwarding 8080 -> 8080: error forwarding port 8080 to pod 216asdfcafebabe123, uid : failed to execute portforward in network namespace "/var/run/netns/cni-01234cafebabe": read tcp4 127.0.0.1:56740->127.0.0.1:8080: read: connection reset by peer
error: lost connection to pod

when curl-ing it with the following:

$ curl -iLk https://localhost:8080/
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8080

then it's something related to forwarding a TLS connection. Forward the http port instead and argocd then works, provided you force it to HTTP only:

$ kubectl -n argocd port-forward svc/argocd-server 8080:80 &
$ argocd login --skip-test-tls --plaintext localhost:8080 
Username: myadminuser
Password:
'myadminuser:login' logged in successfully
Context 'localhost:8080' updated