search

kubernetes / git-sync

A sidecar app which clones a git repo and keeps it in sync with the upstream.
Apache License 2.0
2.21k stars 410 forks source link

V4: GITSYNC_REF or V3: GIT_SYNC_BRANCH #917

Open jorgemgomes opened 1 month ago

jorgemgomes commented 1 month ago

I am experiencing some weird behavior in GitLab CI/CD that I’m not sure if I’m handling correctly.

When I set GITSYNC_REF = CI_COMMIT_REF_NAME (or in V3, GIT_SYNC_BRANCH = CI_COMMIT_REF_NAME), the repository is not cloned unless it's set to the default branch. Does anyone else encounter the same issue when setting the branch to something other than the default branch?

Thanks!

thockin commented 1 month ago

v3 has a broken understanding of the relationship between branches and refs (sorry!). v4 should be saner.

If you're experiencing a problem with v4, I need a simple repro case so I can understand what's happening. Is CI_COMMIT_REF_NAME a branch or tag or SHA? Help me set up a failure case so I can deal wiuth it. I think v4 should be able to do anything, as long as fetch-by-SHA is enabled.

jorgemgomes commented 1 month ago

If I have a job with this configuration:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Values.appName}}-job
  namespace: {{ .Values.namespace }}
  annotations:
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/sync-wave: "-1"
spec:
  template:
    spec:
      volumes:
        - name: dbinit
          emptyDir: { }
      initContainers:
        - image: registry.k8s.io/git-sync:v3.1.6
          name: git-data
          volumeMounts:
            - name: dbinit
              mountPath: /tmp/repo
          env:
            - name: GIT_SYNC_REPO
              value: #######
            - name: GIT_SYNC_ROOT
              value: /tmp/repo
            - name: GIT_SYNC_DEST
              value: "project"
            - name: GIT_SYNC_ONE_TIME
              value: "true"
            - name: GIT_SYNC_USERNAME
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.appName }}-secrets
                  key: gitlabtokenusername
            - name: GIT_SYNC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.appName }}-secrets
                  key: gitlabtokenpassword
          securityContext:
            runAsUser: 0
      containers:
        - image: #######
          name: dbinit          
          resources:
            limits:
              cpu: "500m"
              memory: "256Mi"
            requests:
              cpu: "500m"
              memory: "256Mi"
          volumeMounts:
            - name: dbinit
              mountPath: /deployments
              subPath: project/dbinit
          command: [ "/bin/sh", "-c" ]
          args:
            - ls -l /deployments;
      imagePullSecrets:
        - name: {{ .Values.appName }}-gitlab-docker-auth
      restartPolicy: OnFailure
  backoffLimit: 4

Output of ls -l /deployments is OK:

total 4 drwxr-xr-x 2 root root 54 Aug 13 07:27 changesets -rw-r--r-- 1 root root 378 Aug 13 07:27 db.changelog-master.xml

If I have a job with this configuration:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Values.appName}}-job
  namespace: {{ .Values.namespace }}
  annotations:
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/sync-wave: "-1"
spec:
  template:
    spec:
      volumes:
        - name: dbinit
          emptyDir: { }
      initContainers:
        - image: registry.k8s.io/git-sync/git-sync:v4.2.4
          name: git-data
          volumeMounts:
            - name: dbinit
              mountPath: /tmp/repo
          env:
            - name: GITSYNC_REF
              value: "feature/EUDD-3264-ci-cd-config"
            - name: GITSYNC_REPO
              value: #######
            - name: GITSYNC_ROOT
              value: /tmp/repo
            - name: GITSYNC_DEST
              value: "project"
            - name: GITSYNC_ONE_TIME
              value: "true"
            - name: GITSYNC_USERNAME
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.appName }}-secrets
                  key: gitlabtokenusername
            - name: GITSYNC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.appName }}-secrets
                  key: gitlabtokenpassword
          securityContext:
            runAsUser: 0
      containers:
        - image: #######
          name: dbinit
          resources:
            limits:
              cpu: "500m"
              memory: "256Mi"
            requests:
              cpu: "500m"
              memory: "256Mi"
          volumeMounts:
            - name: dbinit
              mountPath: /deployments
              subPath: project/dbinit
          command: [ "/bin/sh", "-c" ]
          args:
            - ls -l /deployments;
      imagePullSecrets:
        - name: {{ .Values.appName }}-gitlab-docker-auth
      restartPolicy: OnFailure
  backoffLimit: 4

Output of ls -l /deployments is not OK:

total 0

thockin commented 1 month ago

FTR, the diff is:

--- a   2024-08-13 10:07:07.775983193 -0700
+++ b   2024-08-13 10:07:21.943750805 -0700
@@ -13,26 +13,28 @@
         - name: dbinit
           emptyDir: { }
       initContainers:
-        - image: registry.k8s.io/git-sync:v3.1.6
+        - image: registry.k8s.io/git-sync/git-sync:v4.2.4
           name: git-data
           volumeMounts:
             - name: dbinit
               mountPath: /tmp/repo
           env:
-            - name: GIT_SYNC_REPO
+            - name: GITSYNC_REF
+              value: "feature/EUDD-3264-ci-cd-config"
+            - name: GITSYNC_REPO
               value: #######
-            - name: GIT_SYNC_ROOT
+            - name: GITSYNC_ROOT
               value: /tmp/repo
-            - name: GIT_SYNC_DEST
+            - name: GITSYNC_DEST
               value: "project"
-            - name: GIT_SYNC_ONE_TIME
+            - name: GITSYNC_ONE_TIME
               value: "true"
-            - name: GIT_SYNC_USERNAME
+            - name: GITSYNC_USERNAME
               valueFrom:
                 secretKeyRef:
                   name: {{ .Values.appName }}-secrets
                   key: gitlabtokenusername
-            - name: GIT_SYNC_PASSWORD
+            - name: GITSYNC_PASSWORD
               valueFrom:
                 secretKeyRef:
                   name: {{ .Values.appName }}-secrets
thockin commented 1 month ago

Can you please run git-sync with -v 6 and post the logs?

What is "feature/EUDD-3264-ci-cd-config" - is it a branch name, a tag name, or something else?

jorgemgomes commented 1 month ago

feature/EUDD-3264-ci-cd-config -> it is a branch name.

thockin commented 1 month ago

I'll need those logs - git-sync does have tests for syncs against branches.

jorgemgomes commented 1 month ago

How do I run those logs? In initcontainer?

thockin commented 1 month ago

However you normally get logs - e.g. kubectl logs -- please be sure it runs with -v6

jorgemgomes commented 1 month ago

INFO: detected pid 1, running init handler {"logger":"","ts":"2024-08-14 07:21:02.602483","caller":{"file":"main.go","line":568},"level":0,"msg":"starting up","version":"v4.2.4","pid":11,"uid":0,"gid":0,"home":"/tmp","flags":["--add-user=false","--askpass-url=","--cookie-file=false","--credential=[]","--depth=1","--error-file=","--exechook-backoff=3s","--exechook-command=","--exechook-timeout=30s","--git=git","--git-config=","--git-gc=always","--group-write=false","--help=false","--http-bind=","--http-metrics=false","--http-pprof=false","--link=#####","--man=false","--max-failures=0","--one-time=true","--password-file=","--period=10s","--ref=feature/EUDD-3264-ci-cd-config","--repo=#####","--root=/tmp/repo","--sparse-checkout-file=","--ssh-key-file=[/etc/git-secret/ssh]","--ssh-known-hosts=true","--ssh-known-hosts-file=/etc/git-secret/known_hosts","--stale-worktree-timeout=0s","--submodules=recursive","--sync-on-signal=","--sync-timeout=2m0s","--touch-file=","--username=argocd-gitlab-deploy-token","--verbose=6","--version=false","--webhook-backoff=3s","--webhook-method=POST","--webhook-success-status=200","--webhook-timeout=1s","--webhook-url="]} {"logger":"","ts":"2024-08-14 07:21:02.602663","caller":{"file":"main.go","line":669},"level":5,"msg":"running command","cwd":"","cmd":"git version"} {"logger":"","ts":"2024-08-14 07:21:02.604238","caller":{"file":"main.go","line":669},"level":6,"msg":"command result","stdout":"git version 2.39.2","stderr":"","time":"1.505932ms"} {"logger":"","ts":"2024-08-14 07:21:02.604277","caller":{"file":"main.go","line":673},"level":0,"msg":"git version","version":"git version 2.39.2"} {"logger":"","ts":"2024-08-14 07:21:02.604373","caller":{"file":"main.go","line":685},"level":2,"msg":"created private gitconfig file","path":"/tmp/git-sync.gitconfig.532438273"} {"logger":"","ts":"2024-08-14 07:21:02.604418","caller":{"file":"main.go","line":1945},"level":5,"msg":"running command","cwd":"","cmd":"git config --global gc.autoDetach false"} {"logger":"","ts":"2024-08-14 07:21:02.606100","caller":{"file":"main.go","line":1945},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.644155ms"} {"logger":"","ts":"2024-08-14 07:21:02.606131","caller":{"file":"main.go","line":1945},"level":5,"msg":"running command","cwd":"","cmd":"git config --global gc.pruneExpire now"} {"logger":"","ts":"2024-08-14 07:21:02.607402","caller":{"file":"main.go","line":1945},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.235883ms"} {"logger":"","ts":"2024-08-14 07:21:02.607427","caller":{"file":"main.go","line":1945},"level":5,"msg":"running command","cwd":"","cmd":"git config --global credential.helper \"cache --timeout 3600\""} {"logger":"","ts":"2024-08-14 07:21:02.608504","caller":{"file":"main.go","line":1945},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.04684ms"} {"logger":"","ts":"2024-08-14 07:21:02.608531","caller":{"file":"main.go","line":1945},"level":5,"msg":"running command","cwd":"","cmd":"git config --global core.askPass true"} {"logger":"","ts":"2024-08-14 07:21:02.609609","caller":{"file":"main.go","line":1945},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.04356ms"} {"logger":"","ts":"2024-08-14 07:21:02.609637","caller":{"file":"main.go","line":1945},"level":5,"msg":"running command","cwd":"","cmd":"git config --global safe.directory "} {"logger":"","ts":"2024-08-14 07:21:02.610734","caller":{"file":"main.go","line":1945},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.06515ms"} {"logger":"","ts":"2024-08-14 07:21:02.610764","caller":{"file":"main.go","line":1814},"level":1,"msg":"setting up git SSH credentials"} {"logger":"","ts":"2024-08-14 07:21:02.610805","caller":{"file":"main.go","line":1625},"level":3,"msg":"syncing","repo":"#####"} {"logger":"","ts":"2024-08-14 07:21:02.610836","caller":{"file":"main.go","line":1801},"level":1,"msg":"storing git credential","url":"#####"} {"logger":"","ts":"2024-08-14 07:21:02.610864","caller":{"file":"main.go","line":1805},"level":5,"msg":"running command","cwd":"","cmd":"git credential approve"} {"logger":"","ts":"2024-08-14 07:21:02.614480","caller":{"file":"main.go","line":1805},"level":6,"msg":"command result","stdout":"","stderr":"","time":"3.578219ms"} {"logger":"","ts":"2024-08-14 07:21:02.614512","caller":{"file":"main.go","line":1160},"level":3,"msg":"repo directory exists","path":"/tmp/repo"} {"logger":"","ts":"2024-08-14 07:21:02.614528","caller":{"file":"main.go","line":1232},"level":3,"msg":"sanity-checking git repo","repo":"/tmp/repo"} {"logger":"","ts":"2024-08-14 07:21:02.614572","caller":{"file":"main.go","line":1238},"level":3,"msg":"repo directory is empty","path":"/tmp/repo"} {"logger":"","ts":"2024-08-14 07:21:02.614585","caller":{"file":"main.go","line":1168},"level":0,"msg":"repo directory was empty or failed checks","path":"/tmp/repo"} {"logger":"","ts":"2024-08-14 07:21:02.614600","caller":{"file":"main.go","line":1178},"level":0,"msg":"initializing repo directory","path":"/tmp/repo"} {"logger":"","ts":"2024-08-14 07:21:02.614607","caller":{"file":"main.go","line":1179},"level":5,"msg":"running command","cwd":"/tmp/repo","cmd":"git init -b git-sync"} {"logger":"","ts":"2024-08-14 07:21:02.616458","caller":{"file":"main.go","line":1179},"level":6,"msg":"command result","stdout":"Initialized empty Git repository in /tmp/repo/.git/","stderr":"","time":"1.819049ms"} {"logger":"","ts":"2024-08-14 07:21:02.616484","caller":{"file":"main.go","line":1232},"level":3,"msg":"sanity-checking git repo","repo":"/tmp/repo"} {"logger":"","ts":"2024-08-14 07:21:02.616515","caller":{"file":"main.go","line":1243},"level":5,"msg":"running command","cwd":"/tmp/repo","cmd":"git rev-parse --show-toplevel"} {"logger":"","ts":"2024-08-14 07:21:02.617369","caller":{"file":"main.go","line":1243},"level":6,"msg":"command result","stdout":"/tmp/repo","stderr":"","time":"813.013µs"} {"logger":"","ts":"2024-08-14 07:21:02.617398","caller":{"file":"main.go","line":1256},"level":5,"msg":"running command","cwd":"/tmp/repo","cmd":"git fsck --no-progress --connectivity-only"} {"logger":"","ts":"2024-08-14 07:21:02.621140","caller":{"file":"main.go","line":1256},"level":6,"msg":"command result","stdout":"","stderr":"notice: HEAD points to an unborn branch (git-sync)\nnotice: No default references","time":"3.703853ms"} {"logger":"","ts":"2024-08-14 07:21:02.621170","caller":{"file":"main.go","line":1189},"level":5,"msg":"running command","cwd":"/tmp/repo","cmd":"git remote get-url origin"} {"logger":"","ts":"2024-08-14 07:21:02.622191","caller":{"file":"main.go","line":1194},"level":5,"msg":"running command","cwd":"/tmp/repo","cmd":"git remote add origin #####"} {"logger":"","ts":"2024-08-14 07:21:02.623497","caller":{"file":"main.go","line":1194},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.251235ms"} {"logger":"","ts":"2024-08-14 07:21:02.623533","caller":{"file":"main.go","line":1644},"level":3,"msg":"current state","hash":"","worktree":""} {"logger":"","ts":"2024-08-14 07:21:02.623561","caller":{"file":"main.go","line":1750},"level":2,"msg":"fetching","ref":"feature/EUDD-3264-ci-cd-config","repo":"#####"} {"logger":"","ts":"2024-08-14 07:21:02.623580","caller":{"file":"main.go","line":1768},"level":5,"msg":"running command","cwd":"/tmp/repo","cmd":"git fetch ##### feature/EUDD-3264-ci-cd-config --verbose --no-progress --prune --no-auto-gc --depth 1"} {"logger":"","ts":"2024-08-14 07:21:03.079814","caller":{"file":"main.go","line":1768},"level":6,"msg":"command result","stdout":"","stderr":"POST git-upload-pack (461 bytes)\nPOST git-upload-pack (179 bytes)\nFrom #####\n branch feature/EUDD-3264-ci-cd-config -> FETCH_HEAD","time":"456.190767ms"} {"logger":"","ts":"2024-08-14 07:21:03.079883","caller":{"file":"main.go","line":1656},"level":5,"msg":"running command","cwd":"/tmp/repo","cmd":"git rev-parse FETCH_HEAD^{}"} {"logger":"","ts":"2024-08-14 07:21:03.080969","caller":{"file":"main.go","line":1656},"level":6,"msg":"command result","stdout":"952bf88867661007fa9d4a8a2fc67502c8cdda23","stderr":"","time":"1.025849ms"} {"logger":"","ts":"2024-08-14 07:21:03.081013","caller":{"file":"main.go","line":1682},"level":0,"msg":"update required","ref":"feature/EUDD-3264-ci-cd-config","local":"","remote":"952bf88867661007fa9d4a8a2fc67502c8cdda23","syncCount":0} {"logger":"","ts":"2024-08-14 07:21:03.081031","caller":{"file":"main.go","line":1688},"level":5,"msg":"running command","cwd":"/tmp/repo","cmd":"git reset --soft 952bf88867661007fa9d4a8a2fc67502c8cdda23"} {"logger":"","ts":"2024-08-14 07:21:03.082799","caller":{"file":"main.go","line":1688},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.724647ms"} {"logger":"","ts":"2024-08-14 07:21:03.082833","caller":{"file":"main.go","line":1419},"level":1,"msg":"adding worktree","path":"/tmp/repo/.worktrees/952bf88867661007fa9d4a8a2fc67502c8cdda23","hash":"952bf88867661007fa9d4a8a2fc67502c8cdda23"} {"logger":"","ts":"2024-08-14 07:21:03.082853","caller":{"file":"main.go","line":1420},"level":5,"msg":"running command","cwd":"/tmp/repo","cmd":"git worktree add --force --detach /tmp/repo/.worktrees/952bf88867661007fa9d4a8a2fc67502c8cdda23 952bf88867661007fa9d4a8a2fc67502c8cdda23 --no-checkout"} {"logger":"","ts":"2024-08-14 07:21:03.085406","caller":{"file":"main.go","line":1420},"level":6,"msg":"command result","stdout":"","stderr":"Preparing worktree (detached HEAD 952bf88)","time":"2.481159ms"} {"logger":"","ts":"2024-08-14 07:21:03.086068","caller":{"file":"main.go","line":1491},"level":1,"msg":"setting worktree HEAD","hash":"952bf88867661007fa9d4a8a2fc67502c8cdda23"} {"logger":"","ts":"2024-08-14 07:21:03.086102","caller":{"file":"main.go","line":1492},"level":5,"msg":"running command","cwd":"/tmp/repo/.worktrees/952bf88867661007fa9d4a8a2fc67502c8cdda23","cmd":"git reset --hard 952bf88867661007fa9d4a8a2fc67502c8cdda23 --"} {"logger":"","ts":"2024-08-14 07:21:03.095314","caller":{"file":"main.go","line":1492},"level":6,"msg":"command result","stdout":"HEAD is now at 952bf88 EUDD-3264: check logs for git-sync","stderr":"","time":"9.162345ms"} {"logger":"","ts":"2024-08-14 07:21:03.095359","caller":{"file":"main.go","line":1499},"level":1,"msg":"updating submodules"} {"logger":"","ts":"2024-08-14 07:21:03.095378","caller":{"file":"main.go","line":1507},"level":5,"msg":"running command","cwd":"/tmp/repo/.worktrees/952bf88867661007fa9d4a8a2fc67502c8cdda23","cmd":"git submodule update --init --recursive --depth 1"} {"logger":"","ts":"2024-08-14 07:21:03.113145","caller":{"file":"main.go","line":1507},"level":6,"msg":"command result","stdout":"","stderr":"","time":"17.712971ms"} {"logger":"","ts":"2024-08-14 07:21:03.113205","caller":{"file":"main.go","line":1372},"level":2,"msg":"creating tmp symlink","dir":"/tmp/repo","link":"tmp-link","target":".worktrees/952bf88867661007fa9d4a8a2fc67502c8cdda23"} {"logger":"","ts":"2024-08-14 07:21:03.113268","caller":{"file":"main.go","line":1377},"level":2,"msg":"renaming symlink","root":"/tmp/repo","oldName":"tmp-link","newName":"#####"} {"logger":"","ts":"2024-08-14 07:21:03.113313","caller":{"file":"main.go","line":1728},"level":0,"msg":"updated successfully","ref":"feature/EUDD-3264-ci-cd-config","remote":"952bf88867661007fa9d4a8a2fc67502c8cdda23","syncCount":1} {"logger":"","ts":"2024-08-14 07:21:03.113364","caller":{"file":"main.go","line":1213},"level":3,"msg":"cleaning up stale worktrees","currentHash":"952bf88867661007fa9d4a8a2fc67502c8cdda23"} {"logger":"","ts":"2024-08-14 07:21:03.113423","caller":{"file":"main.go","line":1336},"level":4,"msg":"skipping path","path":"/tmp/repo/.worktrees/952bf88867661007fa9d4a8a2fc67502c8cdda23"} {"logger":"","ts":"2024-08-14 07:21:03.113443","caller":{"file":"main.go","line":906},"level":0,"msg":"exiting after one sync","status":0}

thockin commented 1 month ago

That log indicates success. It synced SHA 952bf88867661007fa9d4a8a2fc67502c8cdda23 - is that the wrong SHA for the HEAD of the "feature/EUDD-3264-ci-cd-config" branch of that repo?

If you manually run git clone $REPO -b feature/EUDD-3264-ci-cd-config, what does git rev-parse HEAD show?

What do you mean by "the repository is not cloned unless it's set to the default branch" -- what does happen and what makes you think it is not cloned?

If you run ls -la /tmp/repo, what do you see?

thockin commented 7 hours ago

any updates on this?

jorgemgomes commented 6 hours ago

yes. same situation. just works with the default branch.

thockin commented 5 hours ago

@jorgemgomes I am confused by this reply.

I'm saying that:

But the original bug says it does NOT work.

If you think something is not working, please answer the questions I asked in https://github.com/kubernetes/git-sync/issues/917#issuecomment-2291472434

As it stands, I cannot reproduct this issue.