Repository support using proxy

[doct0r]

Hi there, nice little utility! Unfortunately I would like to try that out in an Openshift cluster that is using a proxy to connect to the Internet, so atm I can't connect to Github. I guess that's not implemented yet, is there any plan to add this? Thanks!

[bigkevmcd]

I think it should work?

It's using the Go standard HTTP client which has support for the environment variables HTTP_PROXY, HTTPS_PROXY and NO_PROXY and lower-cased versions too https://golang.org/pkg/net/http/#ProxyFromEnvironment

Is it not picking up your configuration?

[doct0r]

I tried with a public repo (no auth) and I get timeout errors so I assumed it was because of the proxy. It could be caused by something else but I think it would be quite the coincidence.

[bigkevmcd]

How are you configuring the deployment?

Are you putting the correct env vars into it?

[doct0r]

Thanks for the answer, so I thought this was being picked up by the cluster settings but I was wrong I think. I now tried setting the proxy as env var in the deployment and also as a config map and in both cases I get Proxy Timeout. In any case, I think the operator does work indeed and it's on my side that something is wrong

[bigkevmcd]

Setting this up locally requires a fair bit of effort, but I can do it, and I'l see what I can discover.

[doct0r]

Just another thing to mention, the error I get is when trying to start up the pod, more specifically at step "Trying to become the leader". So after the proxy change in the env, the pod fails to start up properly, that's why I think it's still something on my side.

[bigkevmcd]

Can you paste the startup logs ?

Feel free to mask out anything sensitive, but it would help me to see how far along it's getting.

[doct0r]

Small update, the pod did start successfully after I set the NO_PROXY env var to the same value as the cluster settings. It seems to connect to the repo now but I get a 422 error. I've changed the namespace and repo name in the logs:

{"level":"error","ts":1603713192.20771,"logger":"controller_repository","msg":"Repository poll failed","Request.Namespace":"my-namespace","Request.Name":"my-repository","error":"server error: 422","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/kevin/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/bigkevmcd/tekton-polling-operator/pkg/controller/repository.(*ReconcileRepository).Reconcile\n\ttekton-polling-operator/pkg/controller/repository/repository_controller.go:112\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/Users/kevin/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/kevin/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/Users/kevin/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/kevin/go/pkg/mod/k8s.io/apimachinery@v0.16.5/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/kevin/go/pkg/mod/k8s.io/apimachinery@v0.16.5/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/kevin/go/pkg/mod/k8s.io/apimachinery@v0.16.5/pkg/util/wait/wait.go:88"}

Thanks btw for spending the time to help with this, much appreciated :)

[doct0r]

New update, stupid mistake on my side but I hadn't changed the branch to master, in your example it was main. I now get a 403 but it looks like it's because there is a limit on unauthenticated accounts (I can resolve the API call on my browser at my local machine). I will now try to figure the auth part since I need it anyhow for the actual repo. Thanks a lot for the help!

[bigkevmcd]

Great, let me know if this doesn't help https://github.com/bigkevmcd/tekton-polling-operator#authenticating-against-a-private-repository

[doct0r]

So I have now configured 2 different repositories, one a public repo on Public Github where I do authorise to avoid rate limits and a private one on Enterprise Github where I also authorise. The first one seems to work fine (even though it doesn't trigger the pipeline atm) but for the second one I get a 406 error from Github. Do you know if the operator works with enterprise Github? The URL is <company_name>.github.com and https://api.github.company_name>.com/repos/<org_name>/<repo_name>/commits/master works on Postman.

[bigkevmcd]

It should work, but I tested it it against a company.com installation, rather than a company.github.com installation.

Basically, it takes what you provide as the RepoURL, and attempts to talk to https://host/path if you provide a straight github.com then it will talk to https://api.github.com.

I suspect that if you try https://api.company.github.com/my-org/my-repo.git as a repository URL it will work.

If so, I can change the code to detect this case and simplify it.

[doct0r]

Thanks for the reply, indeed it works with your suggestion 👍 Not sure how common this URL scheme is for Github Enterprise, but it would be a welcome addition I guess :)

I will now try to see if I have the same issue with not starting the pipeline.