`argo archive get` doesn't work with `kubectl port-forward` - `transport: authentication handshake failed`

[saurabh-glam]

Pre-requisites

• [X] I have double-checked my configuration
• [X] I can confirm the issues exists when I tested with :latest
• [X] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Argo CLI command Error:

$ ARGO_SERVER=localhost:2746 argo archive get f4687e41-8781-41d1-870c-4fb06eea1ae3 -n argo
FATA[2023-09-26T09:50:49.377Z] rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake" 

Argo Server is running here:

$ kubectl port-forward -n argo service/argo-workflows-server 2746:2746
Forwarding from 127.0.0.1:2746 -> 2746
Forwarding from [::1]:2746 -> 2746
Handling connection for 2746
E0926 09:50:49.378387 3280561 portforward.go:406] an error occurred forwarding 2746 -> 2746: error forwarding port 2746 to pod c5efb166b1a0c21b3bbe7b79cf05508d13cd61e62b1c6dcd929dddb8da0a6064, uid : failed to execute portforward in network namespace "/var/run/netns/cni-a6a298d2-041f-522c-8e8e-89b47152e769": read tcp4 127.0.0.1:33882->127.0.0.1:2746: read: connection reset by peer
E0926 09:50:49.378606 3280561 portforward.go:234] lost connection to pod

Version

v3.4.11

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

_(any workflow would work - just make sure it's archived)_

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

_(any workflow would work - just make sure it's archived)_

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

_(any workflow would work - just make sure it's archived)_

[terrytangyuan]

Argo server pod is down. Do you see any logs/errors?

[saurabh-glam]

So as soon as I hit enter in the command argo archive get, it kills the argo server port-forward command. Otherwise the argo server is working fine.

[terrytangyuan]

Yeah that's why I am asking the logs for argo server pod so we can find out why it's failed.

[terrytangyuan]

kubectl logs argo-server -p

[m-theisen]

I'm not sure if our setup is close enough to the one reported in this issue, but I temporarily encountered the same issue but setting the ARGO_SECURE flag to false solved the issue for me. So this command should work:

ARGO_SERVER=localhost:2746 ARGO_SECURE=false argo archive get ...

[tooptoop4]

did u try ARGO_SECURE=false @saurabh-glam