search

shipwright-io / build

Shipwright - a framework for building container images on Kubernetes
https://shipwright.io
Apache License 2.0
666 stars 113 forks source link

Proxied Cluster Support #57

Open gabemontero opened 4 years ago

gabemontero commented 4 years ago

today openshift build v1 can obtain certs for the global proxy and ensure they are available to the build process

ultimately buildv2 should have some form of this

it very well might make sense for DEVEX to help abstact out logic used for build v1 for use by buildv2 .... https://github.com/gabemontero/obu is a prototype of such an endeavor

sbose78 commented 4 years ago

today openshift build v1 can obtain certs for the global proxy

Could this be put in a secret and linked to the service account which runs the builds?

gabemontero commented 4 years ago

today openshift build v1 can obtain certs for the global proxy

Could this be put in a secret and linked to the service account which runs the builds?

No that only works with auth @sbose78 not for certs

So what you'll see for the default certs in buildv1 are that configmaps are created to house certs for

I've captured the mechanics of what build v2 will need in obu .... see https://github.com/gabemontero/obu/blob/master/pkg/cmd/cli/cmd/global_proxy_config.go

So ultimately, before calling buildah, you can

Presumably the other build tools also consume the well known proxy env vars and have analogous options to --cert-dir

otaviof commented 4 years ago

What would be the format to specify the proxy settings? Should we expose as a dedicated attribute in Build or are we interested to allow users to control the environment variables?

gabemontero commented 4 years ago

In build v1 we did not surface proxy related items explicitly in the API @otaviof . The build controller just has to always set the env's on the build pod and mount the CA from the config map in the build pod in order for builds to work correctly when OpenShift's global proxy was in play.

The obu link in my previous comment shows how to retrieve that information.

sbose78 commented 4 years ago

@gabemontero The snippet is really helpful, thank you!

A related topic: Where are the trusted CA certs stored in OpenShift / Kubernetes ?

gabemontero commented 4 years ago

@gabemontero The snippet is really helpful, thank you!

A related topic: Where are the trusted CA certs stored in OpenShift / Kubernetes ?

Specific to openshift global proxy, the CAs are in a well known configmap, and there is a seperate controller that can inject those CAs into any CM in any namespace if that CM has a certain label.

In general, CAs are stored in either configmaps or secrets which are mounted in a pod.

By default, the CA for talking to the apiserver is included with the serviceaccount mount for any pod (i.e. every pod has an SA associated with it, and the auth token from that SA is included in the mount, along with the namespace the SA is from (and the pod is running in), the CA for the apiserver.

adambkaplan commented 4 years ago

Tekton may support this feature upstream - it's possible with the right configuration we get this for free.

qu1queee commented 3 years ago

@gabemontero this issue seems to cover two pieces:

  1. Via the usage of ENV variables in the Build, to allow users to define a certificate path, e.g. by passing it into buildah --cert-dir
  2. Support mounting a configmap map in the Build pod, that maps to the above certificate path(one specified by an env variable).

Both points are supposed to apply for certificates for a container registry and for something very Openshift specific(a.k.a global proxy).

We already have issues to address 1) (see https://github.com/shipwright-io/build/issues/48) and 2) (see https://github.com/shipwright-io/build/issues/224). This issue is marked for v1.0.0 release, but seems to be not very clear what exactly needs to be done here, assuming most of the missing pieces are already available in other issues.

Opinions?

gabemontero commented 3 years ago

Your point 1 is not quite right @qu1queee

The env vars are first and foremost for defining the http/https proxy and no proxy env vars so that tools like git perform the necessary changes in their remote interactions.

I agree on your points about #48 and #224 are generic alternatives.

Though it can also be said that proxies are a common enough fact of life for our users that some additional, directed application of #48 and #224 on a global scale based on the presence of a global proxy for a customer could aid cluster administrators. The TEP's in upstream tekton in fact could more or less be categorized along these lines (as is the proxy specific support we provide in openshift).

To that this could be another "keep an item open to monitor tekton in this space" tracker.

If we went down that path, what to do with the fact that this has been assigned to the 1.0.0 GA milestone? I would agree that upstream trackers like this should not necessarily be included in that milestone.

Unless @sbose78 @siamaksade @adambkaplan disagree, I'm good with moving it out of that milestone.

qu1queee commented 3 years ago

@gabemontero ok, +1 on removing this from 1.0.0

sbose78 commented 3 years ago

We do need to verify that standard ways of passing proxy information is respected by the image build operation in a way agnostic to the distribution of Kubernetes.

Example:

Additionally, there could be Build-specific proxy information that could be passed using env vars and volumes in the spec.

I would like to keep this issue around to ensure we at least validate & document how to use global and local proxy information for Builds in Shipwright - even if we don't necessarily have to do anything explicitly :)

Let me know, @qu1queee .

gabemontero commented 3 years ago

We do need to verify that standard ways of passing proxy information is respected by the image build operation in a way agnostic to the distribution of Kubernetes.

Example:

* Globally, the build operation should respect http_proxy env vars ( and other similar ones )

* Globally, the build operation should respect the existence of a proxy certificate in a standard location ( configmap ).

* Globally, Our build tools/strategies should be made aware of the above information.

A bit of an update on this one since @sbose78 last check in beginning of this year (no 9 months ago)

Once both are in place, we can see about say a blog post to introduce the connect with proxy topic, along with probably some additions to our existing API obj markdowns.