feature: support basic auth for private repos

[krancour]

On Slack, @emilwangaa wrote:

in v1 we could set a github token + HTTPS cloneUrl in our project definitions to access private git repositories. In v2 it looks like only SSH keys and git+ssh urls are supported for private repos. Is this a deliberate design decision? Or am I just missing something? (and sorry that I keep bugging you with questions).

This PR addresses this issue.

The feature prompted several modifications to the docs on creating/managing projects, so that section ended up with a comprehensive overhaul. I think they cover the new details well and are more succinct with no real loss of detail.

I will publish a custom build of this branch shortly so @emilwangaa can take this for a test drive.

[krancour]

@emilwangaa, if you'd like to test drive this:

$ export HELM_EXPERIMENTAL_OCI=1
$ export DOCKER_ORG=krancour
$ export TAG=5d0e697

$ helm install brigade oci://ghcr.io/brigadecore/brigade \
    --version v2.4.0 \
    --create-namespace \
    --namespace brigade \
    --set apiserver.image.repository=$DOCKER_ORG/brigade2-apiserver \
    --set apiserver.image.tag=$TAG \
    --set gitInitializer.linux.image.repository=$DOCKER_ORG/brigade2-git-initializer \
    --set gitInitializer.linux.image.tag=$TAG

Preview of updated docs on how to use is here:

https://deploy-preview-1934--brigade-docs.netlify.app/topics/project-developers/projects/#special-secrets-for-working-with-private-git-repos

This custom build also includes the changes from #1935.

[emilwangaa]

@krancour I've tested this and it works fine for Github, but cannot get it to work with Azure DevOps. The vcs init container throws this error:

2022/04/21 12:53:35 Starting Brigade Git Initializer -- version 5d0e697 -- commit 5d0e697
error cloning repository at https://dev.azure.com/<redacted>/<redacted>/_git/first-brigade2-project: unexpected client error: unexpected requesting "https://dev.azure.com/<redacted>/<redacted>/_git/first-brigade2-project/git-upload-pack" status code: 400

Looking through the Azure DevOps documentation it looks like the username:password needs to be a base64 encode string: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Windows#use-a-pat - Do you know if this is done by the git library you're using?

[krancour]

@emilwangaa the base64 encoding you mentioned is just a standard part of basic auth. It's account for. I'll give this a shot.

[krancour]

@emilwangaa ok... I'm getting the same result you are.

Scratching my head at the moment, but we'll figure it out.

[krancour]

@emilwangaa it turns out to be a known issue with the lib we're using: https://github.com/go-git/go-git/issues/64

Well, this is frustrating.

I do think this PR is a net improvement for everyone except ADO users, so I'm not closing, but I'll continue to look at what more we can do.

[emilwangaa]

Good find @krancour, but yeah, that is a bit frustrating 😕

Reading the issue it looks like it also affects ssh when using ADO or do you think that is not the case?

[krancour]

Reading the issue it looks like it also affects ssh when using ADO or do you think that is not the case?

Yes... that's my read on it.

Will open a separate issue for this, but I am looking at the feasability of swapping to git2go, which are the official Go bindings for libgit2. That's a change that would take some time and would have a bit of a ripple effect. We wouldn't be able to compile a statically linked binary anymore, so the git initializer image (for Linux) would get quite a bit larger. I also haven't verified yet that I can easily install libgit2 on a Windows image. (The git initializer component needs both Linux and Windows variants because it could run as an initializer for any job pod, which could be Windows-based.)

i.e. The only reasonable path I see forward is a pretty big change.

[krancour]

@emilwangaa I've mostly got this all worked out... if I can just work out how to make this build for Windows also.

[krancour]

@emilwangaa see WIP PR #1939

This PR still adds a lot of value between, so leaving it open, especially because #1939 borrowed a lot from this one.

[emilwangaa]

@emilwangaa see WIP PR #1939

This PR still adds a lot of value between, so leaving it open, especially because #1939 borrowed a lot from this one.

That looks great @krancour - you work fast👍 I'll gladly take it for a spin when ready.

And yes totally agree on the this PR adding a lot of value, so fine with me if you proceed in merging this in.