Can't clone Git repository from a "dumb" server

[NiklasRosenstein]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

I'm deploying in an on-prem environment and need to make pre-built manifests available to ArgoCD. My chosen method currently involves deploying an additional NGINX web server that serves bare Git repositories (with git update-server-info) from disk.

This works well up until the point when ArgoCD tries to clone the repository.

$ kubectl debug -it -n argocd --image alpine argocd-server-98db887d9-2rd7r -- sh -c '
  apk add git
  git clone http://manifest-server/app-manifests-minio
  ls app-manifests-minio
  head app-manifests-minio/manifests.yaml
'
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
[...]
OK: 18 MiB in 25 packages
Cloning into 'app-manifests-minio'...
manifests.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: minio
  namespace: minio
  labels:
    app.kubernetes.io/instance: minio
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: minio
    app.kubernetes.io/version: 2024.2.13

However, in ArgoCD I get the following error:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = unexpected EOF

To Reproduce

1. mkdir -p /tmp/test && cd /tmp/test && g init --bare . && git update-server-info
2. docker run --name nginx-test -v /tmp/test:/usr/share/nginx/html:ro -p 8080:8080 -d nginx
3. Create an ArgoCD application that reads from http://localhost:8080/ with path: ./

Expected behavior

ArgoCD is able to clone the repository via HTTP and apply the manifest.yaml file from the cloned repository.

Screenshots

Version

argocd: v2.10.0+2175939.dirty
  BuildDate: 2024-02-06T15:31:09Z
  GitCommit: 2175939ed6156ddd743e60f427f7f48118c971bf
  GitTreeState: dirty
  GoVersion: go1.21.6
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.10.1+a79e0ea

Logs

# repo-server
repo-server time="2024-02-21T16:49:50Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2024-02-21T1
6:49:50Z" grpc.time_ms=0.021 span.kind=server system=grpc
# application-controller
time="2024-02-21T16:21:43Z" level=info msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2024-02-21T16:21:43Z\",\"message\":\"Failed to load target state:
 failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = unexpected EOF\",\"type\":\"ComparisonError\"}]}}" application=argocd/minio

[NiklasRosenstein]

Application manifest:

project: default
source:
  repoURL: 'http://manifest-server/app-manifests-minio'
  path: ./
destination:
  server: 'https://kubernetes.default.svc'
  namespace: minio
syncPolicy:
  retry:
    limit: 1

[NiklasRosenstein]

I've now set up HTTPS using a custom root CA, but I end up with the same error after I managed to configure the TLS certificate in ArgoCD (before that it would give x509 certificate signed by unknown authority error as one would expect).

So the issue might not be related to cloning via HTTP instead of HTTPS after all. I've also verified that the repository content can be understood by ArgoCD by replicating the content in a GitHub repository and pulling it from there.

It might be that ArgoCD can't clone from a "dumb Git server". I'm indeed using a dumb NGINX web server at the moment and run git update-server-info on the Git repositories to be able to clone them. I will try to setup up a "Smart HTTP" server next.

[NiklasRosenstein]

Indeed, using fcgiwrap + git-http-backend works with HTTPS. I might give it a try with HTTP tomorrow, but given that I've now got HTTPS set up I might just roll with it.

server {
  server_name _;
  listen 8080 ssl;
  ssl_certificate /opt/ssl/tls.crt;
  ssl_certificate_key /opt/ssl/tls.key;
  ssl_protocols TLSv1.2 TLSv1.3;

  location / {
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
    fastcgi_param GIT_HTTP_EXPORT_ALL "";
    fastcgi_param GIT_PROJECT_ROOT /srv/git;
    fastcgi_param PATH_INFO $uri;
    fastcgi_pass unix:/opt/run/fcgiwrap.socket;
  }
}

Any idea what could be preventing ArgoCD from cloning a Git repository from a "dumb" Git HTTP(S) server?

[mareq]

+1, I am experiencing exactly the same problem

[halvorstein]

Have you tried to overcome this problem by using SSH connection over HTTPS?