Private Repositories bug

[mls-icaro]

Bug description

I am trying to build my Unity Project to Android using the GameCI. My project uses private repositories with ssh. I follow the documentation and I put the SSH Agent step in my CI. I add the Deploy Key to my repository and add the secret key to Secrets.

But this error occurs:

com.company.utils: Error when executing git command. Failed to add the RSA host key for IP address '140.82.112.3' to the list of known hosts (/root/.ssh/known_hosts).
    ERROR: Repository not found.
    fatal: Could not read from remote repository.

I already try to add the ips in /home/runner/.ssh/known_hosts. The command below does not return errors:

Run ssh-keyscan 140.82.114.3 >> /home/runner/.ssh/known_hosts
  ssh-keyscan 140.82.114.3 >> /home/runner/.ssh/known_hosts
  shell: /usr/bin/bash -e {0}
  env:
    SSH_AUTH_SOCK: /tmp/ssh-BYQRT89SNPb3/agent.1664
    SSH_AGENT_PID: 1665
# 140.82.114.3:22 SSH-2.0-babeld-968490c5
# 140.82.114.3:22 SSH-2.0-babeld-968490c5
# 140.82.114.3:22 SSH-2.0-babeld-968490c5
# 140.82.114.3:22 SSH-2.0-babeld-968490c5
# 140.82.114.3:22 SSH-2.0-babeld-968490c5

When I try to add to /root/.ssh/known_hosts, the error occurs:

Run ssh-keyscan github.com >> /root/.ssh/known_hosts
  ssh-keyscan github.com >> /root/.ssh/known_hosts
  shell: /usr/bin/bash -e {0}
  env:
    SSH_AUTH_SOCK: /tmp/ssh-3ZIaGpOiiIJn/agent.1641
    SSH_AGENT_PID: 1642
/home/runner/work/_temp/8d9c2b71-d5e1-4648-b27e-5f695d8e3558.sh: line 1: /root/.ssh/known_hosts: Permission denied
Error: Process completed with exit code 1.

How to reproduce

• Create a unity project that uses private repositories with ssh

• Try to run CI Unity Build

Expected behavior

Build the project with success

CI File

name: Actions 😎

on: [push, pull_request]

jobs:
  build:
    name: Build Project ✨ for ${{ matrix.targetPlatform }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        projectPath:
          - xproject-unity
        unityVersion: 
          - 2020.3.11f1
        targetPlatform:
          - Android
    steps:
      # Checkout
      - name: Checkout repository
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: SSH Agent
        uses: webfactory/ssh-agent@v0.5.3
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

      - name: Adding github to known_hosts
        run: ssh-keyscan github.com >> /root/.ssh/known_hosts

      - name: Adding github to known_hosts
        run: ssh-keyscan 140.82.114.3 >> /root/.ssh/known_hosts

      - name: Adding github to known_hosts
        run: ssh-keyscan 140.82.114.4 >> /root/.ssh/known_hosts

      # Cache
      - uses: actions/cache@v2
        with:
          path: ${{ matrix.projectPath }}/Library
          key: Library-${{ matrix.targetPlatform }}
          restore-keys: |
            Library-${{ matrix.targetPlatform }}-
            Library-

      # Build
      - name: Build project
        uses: game-ci/unity-builder@v2
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
        with:
          sshAgent: ${{ env.SSH_AUTH_SOCK }}
          projectPath: ${{ matrix.projectPath }}
          unityVersion: ${{ matrix.unityVersion }}
          targetPlatform: ${{ matrix.targetPlatform }}
          versioning: Semantic
          androidAppBundle: false
          androidKeystoreName: x.keystore
          androidKeystoreBase64: ${{ secrets.ANDROID_KEYSTORE_BASE64 }}
          androidKeystorePass: ${{ secrets.ANDROID_KEYSTORE_PASS }}
          androidKeyaliasName: ${{ secrets.ANDROID_KEYALIAS_NAME }}
          androidKeyaliasPass: ${{ secrets.ANDROID_KEYALIAS_PASS }}

      # Output
      - uses: actions/upload-artifact@v2
        with:
          name: Build-${{ matrix.targetPlatform }}
          path: build/${{ matrix.targetPlatform }}

[GabLeRoux]

If you have access to sudo, maybe you can try | sudo tee -a like this:

# ...
      - name: Adding github to known_hosts
        run: ssh-keyscan github.com | sudo tee -a /root/.ssh/known_hosts

      - name: Adding github to known_hosts
        run: ssh-keyscan 140.82.114.3 | sudo tee -a /root/.ssh/known_hosts

      - name: Adding github to known_hosts
        run: ssh-keyscan 140.82.114.4 | sudo tee -a /root/.ssh/known_hosts

I just confirmed that this fixes the permission issue you're getting:

https://github.com/GabLeRoux/283-private-repositories/pull/1/files

Before the fix:

After the fix:

🚀

[mls-icaro]

Thanks! I will try this fix and post here the result.

[mls-icaro]

Same error here. how can i access sudo in github workflow?

com.company.utils: Error when executing git command. Failed to add the RSA host key for IP address '140.82.112.4' to the list of known hosts (/root/.ssh/known_hosts).
    ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

[mls-icaro]

The problem is when recovering the private repositories inside UPM.

[GabLeRoux]

com.company.utils: Error when executing git command. Failed to add the RSA host key for IP address '140.82.112.4' to the list of known hosts (/root/.ssh/known_hosts).

ip is 140.82.112.4 in above error message, but the ones you are adding here are different:

      - name: Adding github to known_hosts
        run: ssh-keyscan github.com | sudo tee -a /root/.ssh/known_hosts

      - name: Adding github to known_hosts
        run: ssh-keyscan 140.82.114.3 | sudo tee -a /root/.ssh/known_hosts

      - name: Adding github to known_hosts
        run: ssh-keyscan 140.82.114.4 | sudo tee -a /root/.ssh/known_hosts

I know they're close, but there's a small difference:

140.82.114.3 140.82.114.4 140.82.112.4 <-- the ip from your error

;)

Try also adding this maybe?

      - name: Adding github to known_hosts
        run: ssh-keyscan 140.82.112.4 | sudo tee -a /root/.ssh/known_hosts

[mls-icaro]

In Build Project step

[Package Manager] Done resolving packages in 35.09s seconds
An error occurred while resolving packages:
  Project has invalid dependencies:
    com.company.shareddomain: Error when executing git command. Failed to add the RSA host key for IP address '140.82.114.3' to the list of known hosts (/root/.ssh/known_hosts).
    ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

    com.company.utils: Error when executing git command. Failed to add the RSA host key for IP address '140.82.112.4' to the list of known hosts (/root/.ssh/known_hosts).
    ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

    com.company.missions: Error when executing git command. Failed to add the RSA host key for IP address '140.82.114.3' to the list of known hosts (/root/.ssh/known_hosts).
    ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

UPM is unable to download dependencies. Always returns the same error.

[GabLeRoux]

Ah got it, I think this is because the following commands are not executed in the context of the unityci/editor container:

      - name: Adding github to known_hosts
        run: ssh-keyscan github.com | sudo tee -a /root/.ssh/known_hosts

I think they are executed on github-actions host. At this point, I'm not sure exactly how to execute a command in the container before the build happens. One way to achieve this is probably to do it in the action's code (forking unity builder action and using your own fork which runs these before executing the build).

Something I would try which I really don't know if it will work (that's a shot in the dark):

Just before the build:

      # Build
      - name: Build project
        uses: game-ci/unity-builder@v2

I'd try adding this:

      - name: Adding github to known_hosts in game-ci container
        uses: game-ci/unity-builder@v2
        run: |
          ssh-keyscan github.com | sudo tee -a /root/.ssh/known_hosts
          ssh-keyscan 140.82.114.3 | sudo tee -a /root/.ssh/known_hosts
          ssh-keyscan 140.82.114.4 | sudo tee -a /root/.ssh/known_hosts
          ssh-keyscan 140.82.112.3 | sudo tee -a /root/.ssh/known_hosts
          ssh-keyscan 140.82.112.4 | sudo tee -a /root/.ssh/known_hosts

      - name: Read what's inside known_hosts file in game-ci container
        uses: game-ci/unity-builder@v2
        run: cat /root/.ssh/known_hosts

I did not try this either. Maybe @webbertakken would know better than me here.

[mls-icaro]

Thanks.

Now this error occurs.

Error : .github#L1
a step cannot have both the `uses` and `run` keys

[webbertakken]

As the error says, you either use an action, or you define your own script.

Try running the "run" part before the "uses" part, in separate steps.

[mls-icaro]

Yes, but we are trying to do Unity Builder working with private repositories. I tried the last thing @GabLeRoux said.

We tried to run the run command before the builder, but it looks like it adds the ips in a know_host that is not the Unity docker image.

[webbertakken]

I understand. The idea is that these files get mounted in the container though.

As far as I know you can not run arbitrary commands if you're already assigning your workload to an action. So @GabLeRoux's idea might not work in that exact syntax.

You could however fork builder and add any command you like to try things. Ultimately the container should mount the home folder of the actions runner though.

[mls-icaro]

Thanks.

I know practically nothing about CI. When I fork the unity builder, how will I reference the fork in my CI script? What file do I edit in my fork to add commands?

[mls-icaro]

The IPs always change too.

github.com 140.82.114.3 140.82.114.4 140.82.112.3 140.82.112.4

Now, the ip was 140.82.113.3

What command can I do to add a range of IPs?

[mls-icaro]

After I add all possible IPs, the Know host error is gone.

Now, the new error is:

Project has invalid dependencies:
    com.company.shareddomain: Error when executing git command. ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

    com.company.utils: Error when executing git command. ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

    com.company.missions: Error when executing git command. ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

[GabLeRoux]

The IPs always change too.

github.com 140.82.114.3 140.82.114.4 140.82.112.3 140.82.112.4

Now, the ip was 140.82.113.3

What command can I do to add a range of IPs?

I will answer your questions, but I'm not sure it's exactly the way to go to add all ips on your own. Here's a related stackoverflow question and answers: https://serverfault.com/questions/856194/securely-add-a-host-e-g-github-to-the-ssh-known-hosts-file

I think ssh-keyscan github.com | sudo tee -a /root/.ssh/known_hosts should be good enough.

The IPs always change too.

github ips are documented here:
https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/about-githubs-ip-addresses

They are available here:
https://api.github.com/meta

The ips you are looking for can all be fetched dynamically using a tool like jq command line (see its tldr page):

curl https://api.github.com/meta | jq '.git[]' -r

I'm not 100% sure if known_hosts supports ip ranges per see, but you can definitely resolve this with xargs command line (see its tldr page):

curl https://api.github.com/meta | jq '.git[]' -r | xargs -I {} bash -c "ssh-keyscan {} | sudo tee -a /root/.ssh/known_hosts"

Or in a more readable way/non-one-liner-magical-command using a bash for loop instead of xargs:

meta=$(curl https://api.github.com/meta)
git_ips=$(echo $meta | jq '.git[]' -r)
known_hosts_file=/root/.ssh/known_hosts

for ip in "${arr[@]}"
do
   ssh-keyscan ${ip} | sudo tee -a $known_hosts_file
done

The problem with the above is it won't work. You'll get something like:

getaddrinfo 192.30.252.0/22: nodename nor servname provided, or not known

Because ssh-keyscan doesn't support ip ranges, it only support ips itself. You could probably figure out how to split the ranges into actual IPs, but as I wrote on my first line, I still think you should only need github.com in your known hosts and nothing more.

[GabLeRoux]

After I add all possible IPs, the Know host error is gone.

Now, the new error is:

Project has invalid dependencies:
    com.company.shareddomain: Error when executing git command. ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

    com.company.utils: Error when executing git command. ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

    com.company.missions: Error when executing git command. ERROR: Repository not found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

This is actually the error you want to solve. Looks like you don't have repository read access. Could be your ssh-agent not properly configured or the ssh key being used that is wrong. Ideally, you might want to run the following command in the context of the container to see if things are properly configured:

ssh -v github.com

For example, in my case, I know I'm correctly configured as somewhere in the output, I get the following lines:

Hi GabLeRoux! You've successfully authenticated, but GitHub does not provide shell access.

[GabLeRoux]

When I fork the unity builder, how will I reference the fork in my CI script? What file do I edit in my fork to add commands?

Here are some documentation concerning github actions:
https://docs.github.com/en/actions/creating-actions

1. fork https://github.com/game-ci/unity-builder
2. update your project's yaml file to use your fork instead of upstream action:

      # Build
      - name: Build project
        # uses: game-ci/unity-builder@v2
        # uses: YOUR_USERNAME/unity-builder@v2
        # in your case:
        uses: mls-icaro/unity-builder@v2

v2 refers to a git ref so you could replace v2 with main, push changes to your main branch and run your pipeline again. More details about the syntax can be found here

[mls-icaro]

I will try this and after I will back.

Thanks.