Github checkout fails with status code 128

[aporwal3]

Describe the bug

Git checkout is not working when using pipeline jenkins jobs.

Version of Helm and Kubernetes

- Helm: v3.8.0
- Kubernetes: v1.22.0

Chart version

jenkins-4.2.3

What happened?

Hi, We are using jenkins helmrelease version 4.2.3 and below given is the definition for the same.

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: jenkins-master
  namespace: jenkins-copy
  annotations:
    fluxcd.io/automated: "false"
spec:
  releaseName: jenkins-master
  interval: 10m0s
  timeout: 35m0s
  chart:
    spec:
      chart: jenkins
      sourceRef:
        kind: HelmRepository
        name: jenkins
        namespace: flux-system
      version: "4.2.3"
  values:
    controller:
      componentName: "jenkins-master"
      image: "xxxx.dkr.ecr.us-east-1.amazonaws.com/jenkins"
      tag: "2.361.1-lts-alpine"
      imagePullPolicy: "Always"
      nodeSelector:
        node-class: worker
      podLabels:
        spotinst.io/restrict-scale-down: "true"
      probes:
        startupProbe:
          initialDelaySeconds: 1200
        livenessProbe:
          initialDelaySeconds: 1200
          failureThreshold: 5
        readinessProbe:
          initialDelaySeconds: 1200
          failureThreshold: 5
      hostNetworking: false

      resources:
        requests:
          cpu: "512m"
          memory: "4000Mi"
        limits:
          cpu: "2000m"
          memory: "8000Mi"

      containerEnv:
        - name: AZURE_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: azure
              key: clientId
        - name: AZURE_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: azure
              key: clientSecret
        - name: AZURE_TENANT
          valueFrom:
            secretKeyRef:
              name: azure
              key: tenantId
        - name: DATADOG_JENKINS_PLUGIN_TARGET_API_KEY
          valueFrom:
            secretKeyRef:
              name: datadog
              key: apiKey
        - name: GITHUB_JENKINS_CI_USERNAME
          valueFrom:
            secretKeyRef:
              name: github
              key: username
        - name: GITHUB_JENKINS_CI_API_TOKEN
          valueFrom:
            secretKeyRef:
              name: github
              key: apiToken
        - name: TEST_JENKINS_CI_USERNAME
          valueFrom:
            secretKeyRef:
              name: github-test
              key: username
        - name: TEST_JENKINS_CI_API_TOKEN
          valueFrom:
            secretKeyRef:
              name: github-test
              key: apiToken
        - name: KUBECONFIG_DEVELOPMENT1
          valueFrom:
            secretKeyRef:
              name: kube-config
              key: development1
        - name: KUBECONFIG_PRODUCTION1
          valueFrom:
            secretKeyRef:
              name: kube-config-production1
              key: production1
        - name: CLOUD_ARTIFACTORY_USR
          valueFrom:
            secretKeyRef:
              name: cloud-artifactory-credentials
              key: cloud_artifactory_usr
        - name: CLOUD_ARTIFACTORY_PSW
          valueFrom:
            secretKeyRef:
              name: cloud-artifactory-credentials
              key: cloud_artifactory_psw
        - name: ANYPOINT_USR
          valueFrom:
            secretKeyRef:
              name: anypoint-cli-credentials
              key: anypoint_usr
        - name: ANYPOINT_PSW
          valueFrom:
            secretKeyRef:
              name: anypoint-cli-credentials
              key: anypoint_psw
        - name: SSH_PRIVATE_KEY
          valueFrom:
            secretKeyRef:
              name: devadmin-key
              key: devadmin_service.pem

      javaOpts: "-Xms512m -Xmx4096m"

      disabledAgentProtocols:
        - JNLP-connect
        - JNLP2-connect
        - JNLP3-connect

      # List of plugins to be install during Jenkins master start
      installPlugins:
        - kubernetes:3706.vdfb_d599579f3
        - workflow-aggregator:590.v6a_d052e5a_a_b_5
        - git:4.11.5
        - configuration-as-code:1512.vb_79d418d5fc8  
        - github:1.35.0
        - aws-credentials:latest
        - aws-java-sdk:latest
        - aws-secrets-manager-credentials-provider:latest
        - kubernetes-cd:latest
        - kubernetes-cli:latest
        - kubernetes-credentials:latest

      # Enable to always override the installed plugins with the values of 'master.installPlugins' on upgrade or redeployment.
      overwritePlugins: false
      # Enable HTML parsing using OWASP Markup Formatter Plugin (antisamy-markup-formatter), useful with ghprb plugin.
      # The plugin is not installed by default, please update master.installPlugins.
      enableRawHtmlMarkupFormatter: true
      JCasC:
        enabled: true
        defaultConfig: false
        #pluginVersion: "1.36"
        # it's only used when plugin version is <=1.18 for later version the
        # configuration as code support plugin is no longer needed
        #supportPluginVersion: "1.18"
        configScripts:
          credentials: |
            credentials:
              system:
                domainCredentials:
                    - credentials:
                        - aws:
                            description: "AWS assume role for US-East-1 Management environment"
                            id: "aws-us-east-1-management"
                            scope: GLOBAL
                            iamRoleArn: "arn:aws:iam::569804575345:role/_Development+JenkinsCi"
                        - aws:
                            description: "AWS assume role for US-East-1 Production environment"
                            id: "aws-us-east-1-production"
                            scope: GLOBAL
                            iamRoleArn: "arn:aws:iam::xxxx:role/_Development+JenkinsCi"
                        - usernamePassword:
                            scope: GLOBAL
                            id: github_jenkins_ci_api_token
                            username: "${GITHUB_JENKINS_CI_USERNAME}"
                            password: "${GITHUB_JENKINS_CI_API_TOKEN}"
                            description: GitHub Jenkins CI API Token
                        - usernamePassword:
                            scope: GLOBAL
                            id: test_jenkins_ci_api_token
                            username: "${TEST_JENKINS_CI_USERNAME}"
                            password: "${TEST_JENKINS_CI_API_TOKEN}"
                            description: test Jenkins CI API Token
                        - kubeconfig:
                            description: "kubeconfig for AWS EKS development-1 cluster"
                            id: "development1"
                            kubeconfigSource:
                              directEntry:
                                content: "${KUBECONFIG_DEVELOPMENT1}"
                            scope: GLOBAL
                        - kubeconfig:
                            description: "kubeconfig for AWS EKS production-1 cluster"
                            id: "production1"
                            kubeconfigSource:
                              directEntry:
                                content: "${KUBECONFIG_PRODUCTION1}"
                            scope: GLOBAL
                        - usernamePassword:
                            scope: GLOBAL
                            id: 90db0651-a0bc-46f3-8f68-e52c550b6663
                            username: "${CLOUD_ARTIFACTORY_USR}"
                            password: "${CLOUD_ARTIFACTORY_PSW}"
                            description: Cloud Artifactory Credential (UserName & API key)
                        - basicSSHUserPrivateKey:
                            scope: GLOBAL
                            id: ssh_with_privatekey_provided
                            username: devadmin_service
                            description: "SSH passphrase with private key file. Private key provided"
                            privateKeySource:
                              directEntry:
                                privateKey: ${SSH_PRIVATE_KEY}

      ingress:
        enabled: true
        # For Kubernetes v1.14+, use 'networking.k8s.io/v1beta1'
        apiVersion: "networking.k8s.io/v1beta1" # "extensions/v1beta1"
        annotations:
          kubernetes.io/ingress.class: nginx
        hostName: jenkins-copy.xxx.xxx-xx.com
        tls:
          - secretName: tls-wildcard
            hosts:
              - jenkins-copy.dev.xxx-xxx.com

    agent:
      enabled: false
      image: "xxxx.dkr.ecr.us-east-1.amazonaws.com/jenkins-slave-jnlp-alpine"
      tag: "latest"
      customJenkinsLabels:
        - jenkins-slave
      # name of the secret to be used for image pulling
      imagePullSecretName:
      componentName: "jenkins-slave"
      privileged: false
      resources:
        requests:
          cpu: "512m"
          memory: "512Mi"
        limits:
          cpu: "1000m"
          memory: "2048Mi"
      # You may want to change this to true while testing a new image
      alwaysPullImage: true
      # Controls how agent pods are retained after the Jenkins build completes
      # Possible values: Always, Never, OnFailure
      podRetention: "Always"
      volumes:
      - type: ConfigMap
        configMapName: docker-config
        mountPath: /kaniko/.docker
      - type: EmptyDir
        mountPath: "/tmp"
        memory: false

      command:
      args:
      TTYEnabled: true
      podName: "jenkins-slave"

    persistence:
      enabled: true
      existingClaim: "jenkins-controller"

    serviceAccount:
      create: true
      name: "jenkins-master"
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::xxxx:role/SrevRole-development-1-K8s-JenkinsMaster

    serviceAccountAgent:
      # Specifies whether a ServiceAccount should be created
      create: true
      name: "jenkins-master-agent"
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::xxxx:role/SrevRole-development-1-K8s-JenkinsMasterAgent

***************************************************************************************************************
And now whenever we are running any pipeline job then it fails while checking out github repository with the below error

Error
 > git config remote.origin.url git@github.com:xxx-xxx/orca.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git@github.com:xx-xx/xx.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:1003)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1244)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1308)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:159)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:70)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:311)
    at hudson.model.ResourceController.execute(ResourceController.java:107)
    at hudson.model.Executor.run(Executor.java:449)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git@github.com:xx-xx/xx.git" returned status code 128:
stdout: 
stderr: fatal: not in a git directory

What you expected to happen?

Jenkins pipeline job should successfully be able to checkout the repository.

How to reproduce it

Use the given helmrelease definition to launch the jenkins instance.

Anything else we need to know?

No response

[preetsindhal]

pls, anyone got any fix to this, facing same issue :(

[MishraAbhay]

Facing same issue pls suggest on the same

[jslay-excella]

The issue is the alpine controller image when trying to git checkout a pipeline Jenkinsfile. Only fix I see at the moment is to use centos7 instead.

Anyone from Jenkins able to address this?

[jslay-excella]

After some more digging, this appears to be an issue with persistence on some storage providers.

I had disabled persistence, even set it to an emptyDir, both work with the alpine image. As soon as I enable persistence with our EFS driver, all of our permissions are set for UID and GID 1004 instead of 1000. This seems to cause the alpine image issues, but not the centos7 image, not sure on debian, as our security policies don't allow debian.

I am currently tracking down an issue in our cluster where fsGroup is just ignored when using the EFS CSI driver.

This may be related. https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/125

[aporwal3]

thank you so much @jslay-excella for that advise. We will try to use centos7 and check.

[jslay-excella]

thank you so much @jslay-excella for that advise. We will try to use centos7 and check.

You could also try updating your EFS CSI Driver to anything 1.3.6+, and set the parameters uid and gid on the StorageClass to 1000. Also could switch to another storage class that doesn't use EFS, such as gp2 or gp3, if you have those available. These CSI Drivers should respect securityContext.fsGroup.

These parameters are documented here.

[tudormi]

I can also confirm this issue. The StorageClass used is efs-sc with gidRangeStart: 1000 and gidRangeEnd: 2000, thus the PVC was created with files under ownership 1001,1002, etc But as specified the jenkins user has uid and gid 1000 So I used the solution proposed by @jslay-excella and created a dedicated StorageClass for Jenkins with uid and gid set to 1000 and specified in the Jenkins helm chart and it worked

[OmprakashPaliwal]

I arrived here while investigating https://stackoverflow.com/questions/76279791/jenkins-pipeline-from-scp-throws-stderr-fatal-not-in-a-git-directory/76399342#76399342

When i switched to centos7 based image jenkins/jenkins:lts-centos7-jdk11, Everything works fine without any modification to StorageClass. I was using debian based image earlier so the issue exists in debian image as well.

Any idea what difference centos7 and debian have with EFS?

[MarkEWaite]

I arrived here while investigating https://stackoverflow.com/questions/76279791/jenkins-pipeline-from-scp-throws-stderr-fatal-not-in-a-git-directory/76399342#76399342

When i switched to centos7 based image jenkins/jenkins:lts-centos7-jdk11, Everything works fine without any modification to StorageClass. I was using debian based image earlier so the issue exists in debian image as well.

Any idea what difference centos7 and debian have with EFS?

I don't think that there is any significant different between centos7 and Debian with EFS. The not a git directory message is reported by recent versions of command line git because they have fixed a security issue that was exposed when command line git attempted to operate on a directory that is not owned by the current user. See this Jenkins community discussion for more details.

JENKINS-70540 provides a series of steps that can show the problem without requiring a container image.

The solution is to identify the root problem that is causing the ownership of the directories or files to be different than the current operating system user.

[OmprakashPaliwal]

@MarkEWaite You are right. The reason centos7 image is working is because it's using an old version of git.

$ docker run -it jenkins/jenkins:lts-jdk11 git --version
git version 2.30.2

$ docker run -it jenkins/jenkins:lts-centos7-jdk11 git --version
git version 1.8.3.1

[tareksamni]

If you wish to circumvent the newly introduced security checks (I should point out that they're probably implemented for good reason and it's generally not recommended to disable them 😅), you have the option to execute the following command:

git config --global --add safe.directory "*"

You can run this command within any git repository on the Jenkins master.

Alternatively, you can achieve the same effect by creating a ~/.gitconfig file with the following content:

[safe]
    directory = *

I found that this solution resolved the issues I was encountering with Jenkins, which had been deployed via this helm chart and was running on AWS EKS with EFS mounting.

[tppalani]

HI @tareksamni

I already have entry but still its failing while performing scm checkout, do we need cleanup any workspace inside the master pod where the jenkins is running or what else we can do for fixing this issue, right now we are facing issue in prod environment.

[safe]
    directory = *

[MarkEWaite]

HI @tareksamni

I already have entry but still its failing while performing scm checkout, do we need cleanup any workspace inside the master pod where the jenkins is running or what else we can do for fixing this issue, right now we are facing issue in prod environment.

[safe]
  directory = *

If you've performed that operation on the Jenkins controller and on every Jenkins agent, then you have successfully disabled the security improvement from command line git. That should be enough. I'm not aware of any need for additional cleanup so long as that configuration exists on the Jenkins controller and on every Jenkins agent.

[tppalani]

+

HI @tareksamni I already have entry but still its failing while performing scm checkout, do we need cleanup any workspace inside the master pod where the jenkins is running or what else we can do for fixing this issue, right now we are facing issue in prod environment.

[safe]
    directory = *

If you've performed that operation on the Jenkins controller and on every Jenkins agent, then you have successfully disabled the security improvement from command line git. That should be enough. I'm not aware of any need for additional cleanup so long as that configuration exists on the Jenkins controller and on every Jenkins agent.

What actually this command will do, is that mandatory for git config global level ?

[MarkEWaite]

What actually this command will do, is that mandatory for git config global level ?

The not a git directory message is reported by recent versions of command line git because they have fixed a security issue that was exposed when command line git attempted to operate on a directory that is not owned by the current user. See this Jenkins community discussion for more details.

[tppalani]

What actually this command will do, is that mandatory for git config global level ?

The not a git directory message is reported by recent versions of command line git because they have fixed a security issue that was exposed when command line git attempted to operate on a directory that is not owned by the current user. See this Jenkins community discussion for more details.

You mean to say security issue reported in git version, if yes can you pls tell me what is the version issue has been identified ?

[MarkEWaite]

You mean to say security issue reported in git version, if yes can you pls tell me what is the version issue has been identified ?

I won't do that.

You said earlier that you are affected in production. If you're affected in production, then I think it is quite reasonable that you should read the already linked pages very carefully and understand them thoroughly.

If I read the pages to you, I'm doing you a disservice by not allowing you to read them and understand them yourself. The Jenkins community discussion includes links to the security issue reported against command line git. It outlines the root of the problem and why it is more valuable to solve the root of the problem instead of applying a temporary fix.