actions / runner-images

GitHub Actions runner images
MIT License
9.17k stars 2.84k forks source link

fatal: detected dubious ownership in repository at #6775

Closed AllanOricil closed 1 year ago

AllanOricil commented 1 year ago

Description

I started seeing this error yesterday

image

It does not make sense, because the checkout action is already running the suggested git command to set the repo as safe

image

What could the issue be in this case?

Platforms affected

Runner images affected

Image version and build link

image

Is it regression?

yes. Version: 20221125.1 https://github.com/VodafoneIS/sf-metadata/actions/runs/3601296318/jobs/6066928420

Expected behavior

Since actions/checkout@v3 already run /usr/bin/git config --global --add safe.directory /__w/sf-metadata/sf-metadata I was not expecting to have issues with safe directories.

Actual behavior

image

image

Repro steps

  1. Create a docker container with git installed
  2. Add this image to a job
  3. Run a git command
...

jobs:

  upload-lwc-converage:
    name: Test
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/vodafoneis/salesforce-build-image:main
      credentials:
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

  steps:
      - name: Checkout
        uses: actions/checkout@v3

       - name: run my sh script located in the docker container
         run:  /scripts/do_something_with_git.sh
igorboskovic3 commented 1 year ago

Hi @AllanOricil we will take a look, thanks.

AllanOricil commented 1 year ago

@igorboskovic3 I've set all my workflows to run on ubuntu-20.04 but the same issue happened, maybe because it is using Version: 20221212.2. The last job that ran without a problem was using ubuntu-20.04 20221125.1. So, how can I also set the build version?

igorboskovic3 commented 1 year ago

Hi @AllanOricil, unfortunately you cannot set an old build version, just current one.

AllanOricil commented 1 year ago

@igorboskovic3 I dont understand why we started seeing this issue. This safe dir feature was introduced to git in v2.35.2 https://confluence.atlassian.com/bbkb/git-command-returns-fatal-error-about-the-repository-being-owned-by-someone-else-1167744132.html but the checkout action shows that we are using git v2.20.1 image

Alexey-Ayupov commented 1 year ago

Hello @AllanOricil, we have noticed that the "checkoutv3" action uses a git version that is already installed no matter in the container or in the runner. Below you can find our simple workflow, which went successfully.

jobs:
  run_from_container:
    name: test1
    runs-on: ubuntu-latest
    container:
      image: alpine/git

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: git clone
        run: | 
          git clone --depth=1 https://github.com/actions/runner-images
          ls -la runner-images

  run_from_runner:
    name: test2
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Check the data
        run: ls $GITHUB_WORKSPACE

From this small experiment, you can see that container and runner have different versions. of git. Please pay close attention to your rights in the container and the git version installed in your container as well.

It seems like the issue does not directly relate to a runner, so we will close the issue for now. Feel free to reach us if you have any questions.

AllanOricil commented 1 year ago

@Alexey-Ayupov @igorboskovic3 I did not change anything in any of my workflows since yesterday and this issue started. There is something wrong somewhere and I cant figure out what is it...

AllanOricil commented 1 year ago

@Alexey-Ayupov @igorboskovic3 could this action be the reason? https://github.com/fregante/setup-git-user/blob/master/index.js

I run it after running git checkout. I need it, because inside my workflow I create commits and tags. This action was in my workflow already, and It should not be causing the issue.

AllanOricil commented 1 year ago

I ran this workflow, and again I had the same problem

image

...
jobs:
  validate-changes:
    name: Validate Metadata Changes
    environment: dev
    runs-on: ubuntu-20.04
    container:
      image: ghcr.io/vodafoneis/salesforce-build-image:main
      credentials:
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 2

      - name: Print Git Version
        run: |
          git --version
          touch a.txt
          git add .
          git commit -m "test"
AllanOricil commented 1 year ago

@igorboskovic3 @Alexey-Ayupov tell me it is not a bug somewhere... Look, I added a step Add safe.directory /__w/sf-metadata/sf-metadata and later on the my other git commands fail? Something weird is happening. I insist you reopen this ticket and investigate more. This was not happening, my workflows were working without a problem few days ago without this extra step to set the safe.directory.

image

AllanOricil commented 1 year ago

It is something on my image. I dont know what is it yet, but I just used an image from 7 days ago, and it is working again. will check all my dependencies and then I come back here to tell you what I found.

Alexey-Ayupov commented 1 year ago

Hi @AllanOricil . Thank you for not giving up on searching for root cause. It would be great if you provide your findings.

AllanOricil commented 1 year ago

These are the only diffs between my images.

IMAGE 1 => is the one which I have no issues IMAGE 2 => image where the problem started

image

So, since there aren't any other changes, I have no clue what is causing the problem. Would you like to have access to both images? That may help you to find the real issue.

benzsuankularb commented 1 year ago

@AllanOricil I don't think this is the issue of checkout v3. I started getting the issue today without changing any of my workflows as well. The git version is exactly the same for both the previous successful runner and today's fail runner. No container image build within my runner. Only trying to push a new tag. Did you get a way around it?

AllanOricil commented 1 year ago

@benzsuankularb @Alexey-Ayupov @igorboskovic3 No idea. I just used an older docker image and my problem was gone. Im not sure exactly what is wrong here because there are people reporting the same error and they are not even using docker images. Im just luck, I guess?

In the image below, every commit represents a new container image build. I used the image from b937316 and my issue was gone. Previously, when the problem started, my workflows were using @main.

image

Why did I choose that one? I just picked up the image from the week where all my workflows were not facing that issue.

I just don't know why changing the image fixed my problem, as the only changes between both images are these

image

To prove I did not change anything else on my workflows, but the container images only, here is are some images which you can see the only changes I did on the workflow were the container refs

image

image

This is my docker file:

FROM node:14

RUN apt-get update -y \
 && apt-get upgrade -y \
 && apt-get install -y --no-install-recommends gnupg apt-utils apt-transport-https curl git openssh-client ca-certificates jq bc gettext xmlstarlet bash python3 libxml2 libxml2-utils \
 && rm -rf /var/lib/apt/lists/*

# add an apt repository of chrome
# google-chrome.list will be overwritten by installing google-chrome-stabe
RUN curl -fsSL https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN echo "deb http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list

RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null

# install several packages for CI
RUN apt-get update -y \
  && apt-get install -y --no-install-recommends wget gnupg google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 gh \
  && rm -rf /var/lib/apt/lists/*

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome

COPY setup_npm_global_config.sh /
COPY global_node_dependencies_installer.sh /
COPY package.json /

RUN chmod +x /setup_npm_global_config.sh
RUN chmod +x /global_node_dependencies_installer.sh
RUN /setup_npm_global_config.sh
RUN /global_node_dependencies_installer.sh
RUN sfdx plugins:link /usr/local/lib/node_modules/sfdx-git-delta

ENV SFDX_AUTOUPDATE_DISABLE=false
ENV SFDX_JSON_TO_STDOUT=true
ENV SFDX_USE_GENERIC_UNIX_KEYCHAIN=true
ENV SFDX_DOMAIN_RETRY=true
ENV SFDX_PROJECT_AUTOUPDATE_DISABLE_FOR_PACKAGE_CREATE=true
ENV SFDX_PROJECT_AUTOUPDATE_DISABLE_FOR_PACKAGE_VERSION_CREATE=true
ENV SFDX_DISABLE_DNS_CHECK=true
ENV SHELL /bin/bash

COPY scripts/ scripts/

RUN echo 'y'
RUN sfdx --version
RUN sf --version
RUN sfdx plugins --core
RUN vlocity -v
RUN sfdx sgd:source:delta -h
RUN gh --version
AllanOricil commented 1 year ago

https://github.com/actions/checkout/issues/1048#issuecomment-1356485556

Waed93 commented 1 year ago

I am having the same issue and haven't done any changes. I can see that this issue has been closed with no solution

Waed93 commented 1 year ago

this is fixed by using sudo git config --global --add safe.directory /is/htdocs/wp1090589_IC9AIO06VX/your_project_name it will ask you for the sudo password and there you go :)

AllanOricil commented 1 year ago

this is fixed by using sudo git config --global --add safe.directory /is/htdocs/wp1090589_IC9AIO06VX/your_project_name it will ask you for the sudo password and there you go :)

this isn't a fix for all scenarios

AllanOricil commented 1 year ago

@igorboskovic3 @Alexey-Ayupov could you please reopen this and investigate more. Another random guy reported this in Reddit. https://www.reddit.com/r/github/comments/znfag2/new_dubious_ownership_error_in_github_workflow/

AllanOricil commented 1 year ago

@igorboskovic3 @Alexey-Ayupov I believe the issue comes from this debian updgrade

image

image

Do you have any ideas how I can set this in the dockerfile?

AllanOricil commented 1 year ago

This post also thinks this debian upgrade is the cause of the problem https://phabricator.wikimedia.org/T325128

image

AllanOricil commented 1 year ago

@Alexey-Ayupov @igorboskovic3 I HAVE FINALLY SORTED THIS ISSUE. I built an image which does not upgrade git packages to deb10u6

image

For those that are not using docker images in your workflows, you are probably facing the same issue because the official runners might have upgraded their git package with this security patch. And because you can't change the official runners, your only option is to create a docker image or your own runner, and then ensure the installed git version does not include this security patch mentioned here https://phabricator.wikimedia.org/T325128. In my case, I just had to do the following:

FROM node:14

RUN apt-mark hold git
RUN apt-mark hold git-man
Nicoowr commented 1 year ago

Thanks @AllanOricil ! Another temporary fix is to add the suggested command in the CI/CD pipeline: git config --global --add safe.directory /__w/{repo_name}/{repo_name}

AllanOricil commented 1 year ago

Thanks @AllanOricil ! Another temporary fix is to add the suggested command in the CI/CD pipeline: git config --global --add safe.directory /__w/{repo_name}/{repo_name}

@Nicoowr this does not work when using containers. I tried it already

Nicoowr commented 1 year ago

Got it. It worked in our case though, we're using self-hosted runners with the version 2.299 of Github Actions Runner.

glarez3 commented 1 year ago

Exito peluche!

michaelpeterswa commented 1 year ago

As with @Nicoowr, we encountered this for the first time today on the self-hosted 2.299.1 version of the GitHub Actions Runner. In our case, we were using a python:3.8-buster Docker Container inside the action. We observed that the issue disappeared after switching to a python:3.8-bullseye container which presumably has an updated version of Git. As suggested by @AllanOricil above, try to find a newer container image, currently that seems to be resolving the issue!

More info about the security update here: https://www.debian.org/lts/security/2022/dla-3239

From the above link:

Note: Due to new security checks, access to repositories owned and accessed by different local users may now be rejected by Git; in case changing ownership is not practical, git displays a way to bypass these checks using the new safe.directory configuration entry.

ssbarnea commented 1 year ago

To be honest this should not have being closed because it is 100% related to runners. I see no reason why the permissions should be messed on a runner, it is just bad runner configuration or bad volument mounting for action images.

Yep, it is a side effect of getting a newer git, one that is more sensitive to bad permissions, one that uncovers an issue that was ignored for a very long time.

Related:

FedericoCeratto commented 1 year ago

@ssbarnea setting the permissions correctly should be a better solution. A workaround is to do so manually in the CI when running as root:

     steps:
      - name: Change Owner of Container Working Directory
        run: chown root:root .
AllanOricil commented 1 year ago

@ssbarnea setting the permissions correctly should be a better solution. A workaround is to do so manually in the CI when running as root:

     steps:
      - name: Change Owner of Container Working Directory
        run: chown root:root .

Thanks. I did not thought about it. Does it really work? The downside of this approach is that we must add this step on every workflow in multiple repositories.

yaqwsx commented 1 year ago

I also think this issue shouldn't be closed as, clearly, the permissions are set wrongly. The failing git is only a symptom.

refah7991 commented 1 year ago

i have similar error to yours , but i can't solve? may you helpp me. Screenshot (23)

AllanOricil commented 1 year ago

i have similar error to yours , but i can't solve? may you helpp me. Screenshot (23)

share your workflow here

AllanOricil commented 1 year ago

Problem started again, even holding git packages upgrades.

AllanOricil commented 1 year ago
  • name: Change Owner of Container Working Directory run: chown root:root .

This works if you are running jobs using containers.

Previously I did not have to add these lines. I simply did not upgrade git packages in my images. But now I need this :/ It kind sucks because I have to add it to multiple files.

cedricve commented 1 year ago

Started having this issue as well.. out-of-the-blue..

joelnotified commented 1 year ago

👍 Running into this issue now as well

cedricve commented 1 year ago

@joelnotified if you are installing additional packages in your build one of them might be causing this. What I noticed is that previous "succesful" builds no longer work anymore as well.

danharrin commented 1 year ago

Me too.

Correct output of my action from yesterday - https://github.com/filamentphp/filament/actions/runs/4036577703/jobs/6939267792

The same action today - https://github.com/filamentphp/filament/actions/runs/4043321725/jobs/6952104956

andrew-belac commented 1 year ago

Same error for us

danharrin commented 1 year ago

@Alexey-Ayupov could you reopen this issue please?

imhemish commented 1 year ago

Same issue today after I force pushed my repository, the github pages build failed giving out this error.

jmmclean commented 1 year ago

added this line after my checkout step, which resolved it...but i too started getting this weird error

    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
joelnotified commented 1 year ago

@jmmclean sorry, did that fix the issue or did you get the errors despite that? 🤔

zepfietje commented 1 year ago

@joelnotified @jmmclean, it doesn't fix the issue for us (I'm part of the @filamentphp team with @danharrin).

sjansen-hv commented 1 year ago

We just started seeing this problem today for the first time. For our Docker-based actions the git config needs to run inside the container.

ewjoachim commented 1 year ago

Yup, this is likely to crash all 3rd party container-based actions that do any git work, and are based on an image that has a recent enough version of git. This is what happened to us at https://github.com/py-cov-action/python-coverage-comment-action/issues/127 : suddenly, users all started to report this error.

joelnotified commented 1 year ago

Yeah, I've realized that is the issue as well.

In our case we have a Dockerfile for a custom GitHub Action that uses Git, which I guess has received the latest security updates, causing this.

joelnotified commented 1 year ago

For anyone else with similar issues. I couldn't figure out how to properly set the config from GitPython (which we use within the container), so the easiest solution for us was to put this in the Dockerfile for the custom GitHub Action:

RUN git config --system --add safe.directory *

I realize it's not the best solution, but it works and I don't think there's any risk involved in our case.

AllanOricil commented 1 year ago

For anyone else with similar issues. I couldn't figure out how to property set the config from GitPython (which we use within the container), so the easiest solution for us was to put this in the Dockerfile for the custom GitHub Action:

RUN git config --system --add safe.directory *

I realize it's not the best solution, but it works and I don't think there's any risk involved in our case.

Thanks 😊