Git over ssh behind http_proxy doesn't work with socat proxy from the UI repository page

[Abhishek-Srivastava]

Describe the bug

The git over ssh protocol works fine with till we aren't behind a corporate proxy. To make it work behind the proxy, we need some thirdpart utils like socat and for that we also require to have those extra stanzas in the config file in the .ssh directory in the repo-server container. Something similar to:

Host github.com ProxyCommand socat STDIO PROXY::%h:%p,proxyport=,proxyauth= User git Hostname ssh.github.com Port 443 IdentityFile

Note: Using https option as a workaround forces use of github_token which has wide privilege than a deploy key which is repo specific.

To Reproduce

Configure repo using git protocol behind a corporate proxy. Add socat to the argocd image and mount this config to /home/argocd/.ssh dir

Expected behavior

Repo connection should succeed using the socat proxy

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v1.6.1+159674e
  BuildDate: 2020-06-19T00:39:46Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64

Logs

time="2020-08-18T18:29:44Z" level=error msg="finished unary call with code Unknown" error="dial tcp 140.82.112.36:443: connect: connection timed out" grpc.code=Unknown grpc.method=Create grpc.service=repository.RepositoryService grpc.start_time="2020-08-18T18:27:35Z" grpc.time_ms=128860.26 span.kind=server system=grpc

[jannfis]

The IdentityFile portion of your ssh_config will definitely be overwritten with whatever is configured in ArgoCD (including nothing).

Have you tried setting up the repo in a declarative way, adding the correct SSH key to the declarative config, and just leaving out the ÌdentityFile` parameter completely?

[jannfis]

For being able to add the appropriate repositories using CLI and UI, you could also try to add your ssh_config to the argocd-server pod. This should make it succeed in connecting to your proxied repos.

[Abhishek-Srivastava]

@jannfis Thanks a ton for suggestion and incredible support over the chat. It worked. For others who are facing this issue, I had to do the following:

1. Install socat in argocd docker image
2. Mount a config with on /home/argocd/.ssh/config with:

   Host github.com
   ProxyCommand socat STDIO PROXY:<proxyip>:%h:%p,proxyport=<port>
   User git
   Hostname ssh.github.com
   Port 443

3. Have no_proxy setup properly to exclude the argo services to route via proxy. And voilà, it worked just fine.

[Abhishek-Srivastava]

It turned out that, the cloning works inside the container using the sock proxy but not thru the UI-repository page. It tries connecting to github without the proxy and eventually times out.

time="2020-08-18T18:29:44Z" level=error msg="finished unary call with code Unknown" error="dial tcp 140.82.112.36:443: connect: connection timed out" grpc.code=Unknown grpc.method=Create grpc.service=repository.RepositoryService grpc.start_time="2020-08-18T18:27:35Z" grpc.time_ms=128860.26 span.kind=server system=grpc

[Abhishek-Srivastava]

@jessesuen I you added the tag "help wanted". Do you need any extra information on this bug from me?

[jannfis]

I think the problem is the difference between what's used to test access on the argocd-server (which is git-go and therefore also pure-Go SSH client) and actual repository cloning operations on the argocd-repo-server side (which is, forking git and using openssh).

I have done some research, and I think pure-go implementation of SSH does have proxy support, but will not support any intermediate proxy command. It is able to use a Socks proxy, tho.

I think we should have first-class support for repository access via Proxy (also for HTTPS proxies), instead of using global environment variables defined for the pods (much confusion here -- both, repo server and API server need it, no_proxy needs to be correctly defined, etc etc).

Also, we should really streamline behaviour of Git repository access between the repo server and the API server, because it has been the cause for some nasty surprises in the past. Maybe instead of testing the connection directly from the API server, we could implement a new API method in the repo-server that will test a connection and returns the result to the API server. This would also solve the current issue, I think. I will create an enhancement request and work on an implementation, if there is no good reason to not do it.

[nicc777]

Hi - I am also having this issue. Although my SSH config looks slightly different (using netcat instead of socat) - but for this issue, it's practically identical.

The UI completely ignores the SSH config file and tries to connect directly.

I even tried an approach where I added all the correct files using a custom Docker build.

The SSH config is completely ignored.

[mktgbnk]

@nicc777 were you able to figure it out? We're seeing similar behavior, .ssh/config settings are being ignored

[jannfis]

As a heads-up, I have created https://github.com/argoproj/argo-cd/issues/13166 to introduce proxy support for SSH as a first-class option.