Private git Unity packages fail the compilation process

[Eclextic]

Bug description

Whenever trying to build a Unity project referencing a git package (in UPM), that is privated on GitHub, causes the CI action to fail

Steps to reproduce

1. Create a new Unity Package on GitHub and private the repository
2. Create a Unity project
3. Add the package in the Unity Package Manager using the Git URL
4. Push the Unity Project to git (Launch the action workflow on GitHub/Build the project)

I can't really as I can't configure a Git credentials helper

Expected behavior

Just build normally (Might have to add git credentials for that to work, somehow...)

Additional details

[bardic]

You need to have a key in your pipeline env vars in your repo settings, the public in your account keys and then pass the private key env var to your pipeline so that it can keyscan github and had it to your known_hosts.

I don't have a github snippet but this is what I go on gitlab

if ! [[ -z "${SSH_PRIVATE_KEY}" ]]; then
  eval $(ssh-agent -s) && \
  echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - && \
  mkdir -p ~/.ssh && \
  chmod 700 ~/.ssh && \
  ssh-keyscan $CI_SERVER_HOST >> ~/.ssh/known_hosts && \
  chmod 644 ~/.ssh/known_hosts
fi 

[Eclextic]

You need to have a key in your pipeline env vars in your repo settings, the public in your account keys and then pass the private key env var to your pipeline so that it can keyscan github and had it to your known_hosts.

I don't have a github snippet but this is what I go on gitlab

if ! [[ -z "${SSH_PRIVATE_KEY}" ]]; then
  eval $(ssh-agent -s) && \
  echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - && \
  mkdir -p ~/.ssh && \
  chmod 700 ~/.ssh && \
  ssh-keyscan $CI_SERVER_HOST >> ~/.ssh/known_hosts && \
  chmod 644 ~/.ssh/known_hosts
fi 

If possible could I get some guidance? I'm first of all pretty new to CI and am trying building something in CI for the first time.

What I did:

I've created a clean project with only one UPM git package installed. Without the package, the CI works/builds perfectly. With it we get the same error as above!

I've tried using the git-credential manager from this site https://www.jayconrod.com/posts/114/accessing-private-github-repositories-over-http but it somehow didn't work (I of course removed everything with go-lang and only kept the authentication step)

Inside of Unity I use the HTTPS protocol for the git packages so to follow what you suggested I tried to convert them to use the SSH-based protocol, but Unity didn't allow it? (Read the documentation on their site too: https://docs.unity3d.com/Manual/upm-git.html#Git-SSH) Tried both formatting styles, the one on their site and the one copied directly from GitHub...

• GitHub Format: [git@github.com:TangerineGameDev/CLog.git]()
• Unity UPM Format: [ssh://git@TangerineGameDev.github.com/CLog/com.tangerinedev.clog.git]()

So I abandoned your idea as Unity wouldn't allow me to use SSH.

I honestly am pretty exhausted and don't know what do anymore...

[Eclextic]

Just know I found this article: https://phoenixnap.com/kb/ssh-with-key#:~:text=Public%20Key%20Authentication%20is%20a,strength%20and%20automated%20passwordless%20logins.

Would you say it's worth pursuing? (Of course as long as I find a way to get SSH working in Unity)

[bardic]

Hey, Sorry for the delay. So in your UPM manfiest.json you want to reference a private repo like this :

"com.project.example": "git@gitlab******.com:project/example.git",

For getting access to that repo on github you can use this action:

https://github.com/marketplace/actions/webfactory-ssh-agent

or the method I mentioned earlier but will explain it bit more :

After adding the repo to your manifest.json you then need make a private/public key on your local machine.

ssh-keygen -t ed25519 -C "your_email@example.com"

Then in your repo that you want to be able to access your private one, add the ssh key as a private env var.

And now is the tricky part but that is mostly due to my ignorance. I use gitlab and have access to the runners and users on them so adding my public key was easy.

But I know there's a method called deploy keys to help with that : https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys

After you get your pub key added somehow, that script I posted earlier which I based ( read copied) from here : https://gitlab.com/gitlab-examples/ssh-private-key/-/blob/master/.gitlab-ci.yml

if ! [[ -z "${SSH_PRIVATE_KEY}" ]]; then
  eval $(ssh-agent -s) && \
  echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - && \
  mkdir -p ~/.ssh && \
  chmod 700 ~/.ssh && \
  ssh-keyscan $CI_SERVER_HOST >> ~/.ssh/known_hosts && \
  chmod 644 ~/.ssh/known_hosts
fi 

Checks for an env var named SSH_PRIVATE_KEY.

If it exists it'll start ssh, fix the key line endings, add it, make all the dir and permissions and then scan the host of your private repo so that it's added to your known hosts and then fix its permissions.

After that you can launch unity and it'll be able to access the repo. We do this often in our company for sharing packages and is a pain in the ass to get started but afterwards is pretty slick.

[Eclextic]

Hey, Sorry for the delay. So in your UPM manfiest.json you want to reference a private repo like this :

"com.project.example": "git@gitlab******.com:project/example.git",

For getting access to that repo on github you can us this action:

https://github.com/marketplace/actions/webfactory-ssh-agent

or the method I mentioned eaelier but will explain it but more :

After adding the repo to your manifest.json you then need make a private/public key on your local machine.

ssh-keygen -t ed25519 -C "your_email@example.com"

Then in your repo that you want to be able to access your private one, add the ssh key as a private env var.

And now is the tricky part but that is mostly due to my ignorance. I use gitlab and have access to the runners and users on them so adding my public key was easy.

But I know there's a method called deploy keys to help with that : https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys

After you get your pub key added somehow, that script I posted earlier which I based ( read copied) from here : https://gitlab.com/gitlab-examples/ssh-private-key/-/blob/master/.gitlab-ci.yml

if ! [[ -z "${SSH_PRIVATE_KEY}" ]]; then
  eval $(ssh-agent -s) && \
  echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - && \
  mkdir -p ~/.ssh && \
  chmod 700 ~/.ssh && \
  ssh-keyscan $CI_SERVER_HOST >> ~/.ssh/known_hosts && \
  chmod 644 ~/.ssh/known_hosts
fi 

Checks for an env var named SSH_PRIVATE_KEY.

If it exists it'll start ssh, fix the key line endings, add it, make all the dir and permissions and then scan the host of your private repo so that it's added to your known hosts and then fix its permissions.

After that you can launch unity and it'll be able to access the repo. We do this often in our company for haring packages and is a pain in the ass to get started but afterwards is pretty slick.

Thank you very much for the verbose answer! I‘ll try it later today!

[Eclextic]

Honestly I'm still having problems man @bardic...

What worked/What I did

Let's go!

So first of all, I couldn't use SSH to add a package in Unity.

• FIX: Create my own SSH Key (priv/pub) into a directory called C://User/.ssh using ssh-keygen and also add the host (github.com) to a file called "known_hosts" in the same directory. https://stackoverflow.com/a/29380765. I generated every encryption value (RSA, ECDSA and ED25519).

Now Unity accepts SSH packages, but only in this format ⇒ [git@github.com:username/package.git]()

---

Now I need to add the SSH information into the docker, but we still have problems...

The Problem

Now we have "Host Key Verification Failed", even though I am sure I added the host name.

This error only pops up whenever you don't have the "know_hosts" file, but it exists using ls on the server and gives me the same encrypted value as in my own PC (I assume this is a good sign, as it is GitHub's public key)

Whenever one doesn't have their on encrypted key a different error is given...

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

So as you can guess, I am irritated... The "know_hosts" file exists on the server-side, but the error that the server-side is giving me, literally only occurs whenever you have NO "know_hosts" file or nothing in the directory AT ALL.

Help!

[Eclextic]

So it actually seems to work now with this setup:

name: Build project

on: [push, pull_request, workflow_dispatch]

jobs:
   buildForAllSupportedPlatforms:
      name: Build for ${{ matrix.targetPlatform }}
      runs-on: ubuntu-latest
      strategy:
         fail-fast: false
         matrix:
            targetPlatform:
               #- StandaloneOSX # Build a macOS standalone (Intel 64-bit).
               - StandaloneWindows # Build a Windows standalone.
               #- StandaloneWindows64 # Build a Windows 64-bit standalone.
               #- StandaloneLinux64 # Build a Linux 64-bit standalone.
               #- iOS # Build an iOS player.
               #- Android # Build an Android .apk standalone app.
               #- WebGL # WebGL.
      steps:
         - uses: actions/checkout@v3
           with:
              fetch-depth: 0
              lfs: true

         - uses: webfactory/ssh-agent@v0.8.0
           with:
              ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

         - name: Configure Private Repo Access
           run: |
              mkdir -p ~/.ssh
              chmod 700 ~/.ssh
              touch ~/.ssh/known_hosts
              ssh-keyscan -t rsa github.com | tee github-key-temp | ssh-keygen -lf -
              cat github-key-temp >> ~/.ssh/known_hosts
              ssh-keyscan -t ed25519 github.com | tee github-key-temp | ssh-keygen -lf -
              cat github-key-temp >> ~/.ssh/known_hosts
              ssh-keyscan -t ecdsa github.com | tee github-key-temp | ssh-keygen -lf -
              cat github-key-temp >> ~/.ssh/known_hosts
              ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts

         - uses: actions/cache@v2
           with:
              path: Library
              key: Library-${{ matrix.targetPlatform }}
              restore-keys: |
                 Library-Test-
                 Library-

         - uses: game-ci/unity-builder@v2
           env:
              UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
           with:
              targetPlatform: ${{ matrix.targetPlatform }}
              sshAgent: ${{ env.SSH_AUTH_SOCK }}
              allowDirtyBuild: true

         - uses: actions/upload-artifact@v2
           with:
              name: Build-${{ matrix.targetPlatform }}
              path: build/${{ matrix.targetPlatform }}

Keep in mind that the public SSH Key was made to be a deploy key in the repository I want to pull from.

But only using one key, where I want multiple... I've seen that there is something like a config file for SSH where you can use specific key files (%USERPROFILE%/.ssh directory) and point them to certain links ([git@github.com:username/package.git]()) where instead of writing out the whole git user name and repository it just becomes a simple link. Some Example

This could work, but seems pretty painful to manage...

There also exists something called SSH agent forwarding which I might should research...

Honestly I might just do SSH Config as my project currently is a simple one, but I would be open to suggestions!

[Eclextic]

I am currently pretty busy, so I’ll close it for now, but I’ll come back to it later…