Image Pull Secret and Proxy Support

[coney]

In our company network, we cound only access internet via a internal proxy, and all certificates are self signed. Need to change the go server & agent images to add custom ca and add proxy environments. Besides, our self hosted docker registry require authentication. and I need to manually update the deployment to specified pulling secret.

Request for the following capabilities for helm chart:

1. expose pulling secret options.
2. able to set proxy(and no proxy configs) via helm values
3. able to add custom ca certs

[chadlwilson]

A PR for image pull secret, or similar chart to model off would be welcome.

For which functions do you expect the proxy and custom CA cert to be used for? It's already possible to set arbitrary env vars with server.env.extraEnvVars if it's just for pre-startup plugin installs.

If you need regular server operations (polling materials or repositories) to go through the proxy, you might not have much luck based on https://github.com/gocd/gocd/issues/8699 - and I suspect it would be misleading to add special config to the Helm chart for functionality that the server does not really support. Agents support being proxied via https://docs.gocd.org/current/installation/configure-agent-proxy.html which should be able to be configured with agent.env.goAgentJvmOpts.

If you already have a solution/workaround/hack that works for you, please share it so we can see if it's appropriate to make more general.

[coney]

Our proxy and internal website uses a self signed certificate which I need to ignore the ssl verification or add our root CA to trusted store.

For the custom CA issue, I created a docker image based on GoCD's official image and manually appended to /etc/ssl/certs/ca-certificates.crt. And specified the gocd.server.image to the new one. But this is definitely not a general solution. May be the helm chart could add some arguments which allows to append custom CA contents(maybe based64 encoded) to /etc/ssl/certs/ca-certificates.crt before server starts.

And for the pull secret, I patches the gocd-server deployment after installs/upgrads by helm using kubectl -ngocd patch deploy gocd-agent -p '{"spec":{"imagePullSecrets":[{"name":"regcred"}]}}'. the secret regcred is created by myself and contains credential against the private registry.

[chadlwilson]

For the custom CA issue, I created a docker image based on GoCD's official image and manually appended to /etc/ssl/certs/ca-certificates.crt. And specified the gocd.server.image to the new one. But this is definitely not a general solution. May be the helm chart could add some arguments which allows to append custom CA contents(maybe based64 encoded) to /etc/ssl/certs/ca-certificates.crt before server starts.

OK, so you only care that GoCD can start up and bootstrap plugins? This bit generally does require internet access if you want it to auto-install plugins from GitHub. Are you aware that this behaviour can be disabled via server.shouldPreconfigure=false

And to confirm, you don't need the GoCD Server itself to use your custom CA cert? (because I don't believe that will work with /etc/ssl/certs/ca-certificates.crt alone)

If you do want the downloaded plugins, you already have these options that I can see

• you can already mount your own ca-certificates.crt from a volume/configmap using persistence.extraVolumeMounts without building a new image
• or if you would prefer to build a Docker image, you can pre-include the relevant plugins during docker build so there is no runtime dependency to talk out to the internet/GitHub and disable download with server.shouldPreconfigure=false
• or alternatively to put the plugins on some storage and mount a volume that tells GoCD how to find them

[brettcurtis]

I'd like to see support for helm static agents as well using :

spec:
  imagePullSecrets:
  - name: github-container-registry-key

We can do this with the elastic agents, and it works well, I'd like to be able to edit my values file to achieve the same on the static agent.

[brettcurtis]

Here is a good example I think: https://github.com/Azure-Samples/helm-charts/blob/master/chart-source/image-pull-secret/templates/acs-helloworld.yaml

I've never done anything with helm templates, but I'll see if I can figure it out and submit a PR.

[chadlwilson]

For the record, I don't have any particular concern with imagePullSecrets, however I am much more suspicious about attempts to deal with proxies.

There are other ways to approach thiswhich the chart already supports (above, even if they could be better documented) and other attempts in the chart seem likely to give a misleading impression as to whether it is configuring overall application proxy support (which is a somewhat different problem).

[coney]

For the custom CA issue, I created a docker image based on GoCD's official image and manually appended to /etc/ssl/certs/ca-certificates.crt. And specified the gocd.server.image to the new one. But this is definitely not a general solution. May be the helm chart could add some arguments which allows to append custom CA contents(maybe based64 encoded) to /etc/ssl/certs/ca-certificates.crt before server starts.

OK, so you only care that GoCD can start up and bootstrap plugins? This bit generally does require internet access if you want it to auto-install plugins from GitHub. Are you aware that this behaviour can be disabled via server.shouldPreconfigure=false

And to confirm, you don't need the GoCD Server itself to use your custom CA cert? (because I don't believe that will work with /etc/ssl/certs/ca-certificates.crt alone)

If you do want the downloaded plugins, you already have these options that I can see

• you can already mount your own ca-certificates.crt from a volume/configmap using persistence.extraVolumeMounts without building a new image
• or if you would prefer to build a Docker image, you can pre-include the relevant plugins during docker build so there is no runtime dependency to talk out to the internet/GitHub and disable download with server.shouldPreconfigure=false
• or alternatively to put the plugins on some storage and mount a volume that tells GoCD how to find them

The certificate is not only for GoCD to bootstrap and download plugins, after GoCD started, the certifacte is also required to pull the self-hosted git server which using the self signed certs.

[coney]

I'd like to see support for helm static agents as well using :

spec:
  imagePullSecrets:
  - name: github-container-registry-key

We can do this with the elastic agents, and it works well, I'd like to be able to edit my values file to achieve the same on the static agent.

For elastic agent it's easy to configure pull secrets, but since I need to create a custom GoCD server image, I wish I cound provide the pull secret in the helm arguments. Besides, since GoCD helm chat provides the capability to custom the image repository, I think it should provide the way to configure pull secret as well(Offical Docker hub now have quotas and ratelimit for pulling the image, even if I'm not creating the custom GoCD server images, I need to mirror the server image from office Docker hub to my private registry which enables authentication)

[chadlwilson]

The certificate is not only for GoCD to bootstrap and download plugins, after GoCD started, the certifacte is also required to pull the self-hosted git server which using the self signed certs.

@coney This only works by accident, because you happen to use git, and GoCD executes git commands by forking the native git binary, which in turn happens to use libcurl which happens to also default to loading certs from /etc/ssl/certs/ca-certificates.crt. Any change to these dependencies would break a special solution in the Helm chart that alters /etc/ssl/certs/ca-certificates.crt, and thus it is not appropriate for the chart to have special logic that assumes the behaviour of these lower layers.

It will also likely not work if proxy is required for other source control materials, or for some artifact repository which uses a native Java integration via a plugin, or perhaps for an elastic agent plugin that may also need to talk through a proxy.

• you can already mount your own ca-certificates.crt from a volume/configmap using persistence.extraVolumeMounts without building a new image

As in the above suggestion, if your hack/custom /etc/ssl/certs/ca-certificates.crt already works for you, the helm Chart already supports mounting anything you like there, so as far as I can tell there is no requirement for a change to the chart.

To add fuller example, create or modify ca-certificates however you feel like

1. kubectl create configmap gocd-server-ca-certs --from-file=ca-certificates.crt
2. Modify values and helm update

   server:
   env:
   extraEnvVars:
   - name: https_proxy
     value: http://proxy.company.com
   - name: no_proxy
     value: internal-thing.company.com
   persistence:
   extraVolumes:
   - name: gocd-server-ca-certs
     configMap:
       name: gocd-server-ca-certs
       defaultMode: 0755
   extraVolumeMounts:
   - name: gocd-server-ca-certs
     mountPath: /etc/ssl/certs/ca-certificates.crt
     subPath: ca-certificates.crt
     readOnly: true

The same thing can be done for static agents provided by the chart, or in kubernetes elastic agent configuration.

We should not add configuration to the Helm chart which is misleading as to the capability it provides across the many things GoCD can do, and your specific use case of customising the ca-certs seems already supported. Please let me know if you have tried the above.

Supporting imagePullSecrets makes sense, will merge something shortly.

[chadlwilson]

The support for imagePullSecrets will be available in 1.40.1 shortly. Thanks @brettcurtis for getting a start on this in https://github.com/gocd/helm-chart/pull/35

I'm going to close this issue now, as I feel the chart already has adequate support for customising ca-certificates.crt and proxy env vars for use by curl/libcurl as noted above, without needing to build a custom Docker image. If this doesn't work for you @coney please provide a bit more information and we can open a more specific issue.