search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
301 stars 115 forks source link

cf-for-k8s should support a great and easy first time / trial experience #265

Open cmoulliard opened 4 years ago

cmoulliard commented 4 years ago

Feature request

Using ytt or kapp tools are not really easy excepted if you maintain the cf-for-k8s project. Users, interested to play with cf-fork8s, dont have necessarily the time to learn another K8s tools as most of them knowhelm(which is a k8s and cncf standard) or justkubectl apply -f`.

Here is by example a typical error that you will get if you try to change a parameter as currently we ask to the user to generate from templates the k8s yaml files

ytt -f config -f /tmp/cf-values.yml > /tmp/cf-for-k8s-rendered.yml
ytt: Error: Overlaying data values (in following order: values.yml, cf-values.yml):
  Document on line cf-values.yml:2:
    Map item (key 'images') on line cf-values.yml:192:
      Map item (key 'cf_autodetect_builder') on line cf-values.yml:193:
        Expected number of matched nodes to be 1, but was 0

Without strong and deep knowledge about ytt and also cf-for-k8s, it will be, for most of users, impossible to fix the problem as the solution implies that you know how templates have been created, are linked, ...

This is the reason why I would like to recommend that you initiate a discussion about how cf-for-k8s will be consumed/used by the users and have a great trial experience. One idea could be to convert the existing files into heml charts

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/173602146

The labels on this github issue will be updated when the story is started.

aegershman commented 4 years ago

I apologize in advance because I don't want to diminish this logged issue or speak in any way to undermine your request. I'm not an authority by any means obviously, I'm just some dude™️; nonetheless-- I'd like to say we've had pleasant success with ytt for cf-for-k8s as well as other tooling such as concourse pipelines. We've found ytt files, especially overlays, are like bosh operator, but even better. We've found them to be easier to use, extensible, and provide faster feedback loops than helm. With ytt I can provide my own operations which can add/remove/modify contents for the final rendered product. With helm, you can't modify anything about the upstream templates, you only have access to provide values to what the maintainer has parameterized in gotmpl. With plain yaml + ytt, not only can you provide values to the rendered product, but if the situation warrants, you can provide your own templating modifiers. Which in a sufficiently complex (and thus parametrizable) system is quite valuable. We've also found that it's much easier to distribute "opsfiles" (in the form of ytt files with overlays and definitions and values and such) with ytt than with helm. I'm hypothesizing but I think this is partially the reason why certain projects end up dropping helm in favor of their own templating packaging, like istioctl, kiali, etc., all having their native config generation model being specific like istioctl manifest generate. As time goes on I think more practical examples will emerge of use-cases for user-supplied (or more optional vendor-supplied?) ytt files (especially overlays) will be useful for those who are using ytt for the first time. Again, that's just my take 👍

mike1808 commented 4 years ago

(I also apologize as I am not a maintainer for this repo). I understand the frustration of the issue author about unfriendly ytt error messages, however, as was mentioned earlier by @aegershman above ytt is a very powerful tool compared to Helm.

I wonder what if ytt had more friendly error messages which will help and guide you to troubleshoot errors. Will it make playing with cf-for-k8s easier?

cmoulliard commented 4 years ago

I wonder what if ytt had more friendly error messages which will help and guide you to troubleshoot errors. Will it make playing with cf-for-k8s easier?

There are fundamentally different issues to use ytt and kapp as :

cmoulliard commented 4 years ago

I apologize in advance because I don't want to diminish this logged issue or speak in any way to undermine your request. I'm not an authority by any means obviously, I'm just some dude™️; nonetheless-- I'd like to say we've had pleasant success with ytt for cf-for-k8s as well as other tooling such as concourse pipelines. We've found ytt files, especially overlays, are like bosh operator, but even better. We've found them to be easier to use, extensible, and provide faster feedback loops than helm

ytt is perhaps and certainly a great tool for the engineers responsible the assemble the needed templates and/or k8s yaml resources for cf-for-k8s but not for the lambda users interested to install easily cf-for-k8s. No need to install ytt, kapp or even bosh client tool (to just populate certificates !!!!) but only helm + kubectl in order to start a great developer experience with cf on kind kubernetes.

loewenstein commented 4 years ago

Disclaimer 1: I am also not a maintainer of cf-for-k8s. Disclaimer 2: I am sceptical about the usage of a tool that is open source, but lacks widespread usage in the broader Kubernetes community. We had (well, have for current mainstream CF) such a tool: BOSH. It is really great in what it does, but it lacks adoption outside the CF community. I do see a risk that something similar will happen to the k14s tool suite. I wouldn't go as far as saying that the fate is decided yet though.

With that being said, I don't think Helm is a great tool for maintaining systems based on Kubernetes, despite its widespread usage in the community. The aforementioned example of Istio using there own tooling to create installation manifests is a good indicator that Helm lacks certain capabilities for production grade installations.

I do suppose the title of this issue is potentially misleading due to a mix of problem and solution space. Could we instead say: "cf-for-k8s should support a great and easy first time / trial experience" and still capture the gist of what you would like to see happening, @cmoulliard?

What if cf-for-k8s, as part of its releases, would publish a fully rendered Kubernetes manifest for installing CF onto e.g. a KinD cluster. Would that cover the use case you have in mind? Skimming over the cf-for-k8s default values file it should be possible to provide defaults for everything except maybe the docker registry. I am sure we could find an easy way to provide that without requiring templating.

Somewhat related: As far as I know the Eirini team is currently looking into a template free installation base. If this model gets traction in the community and gets adopted by (all) other cf-for-k8s components, it should be easy for any interested party to provide a Helm based "distribution" of CF@K8S.

cmoulliard commented 4 years ago

I do suppose the title of this issue is potentially misleading due to a mix of problem and solution space. Could we instead say: "cf-for-k8s should support a great and easy first time / trial experience" and still capture the gist of what you would like to see happening, @cmoulliard?

I agree with you that the original title and how discussions took place is not longer accurate and should be renamed. I will review it ;-)

cmoulliard commented 4 years ago

Disclaimer 2: I am sceptical about the usage of a tool that is open source, but lacks widespread usage in the broader Kubernetes community. We had (well, have for current mainstream CF) such a tool: BOSH. It is really great in what it does, but it lacks adoption outside the CF community. I do see a risk that something similar will happen to the k14s tool suite. I wouldn't go as far as saying that the fate is decided yet though.

Using and adopting open tools should be the norm, standard ;-)

I don t say at all that this project should abandon to use ytt, kapp or bosh for the work which is needed to maintain the template able to generate the k8s yaml resources of cf BUT they should only be used by the cf-for-k8s developers

The users interested to play with cf-for-k8s on k8S to experiment a great developer experience should rely on Helm and why not on a cf client command able to install on k8s CloudFoundry -> cf setup --cf config.yml

Birdrock commented 4 years ago

@cmoulliard Thank you for bringing your concerns to us. It seems like the discussion should be moved to the #cf-for-k8s slack channel in the cloudfoundry slack. If a more decisive plan comes out of community discussion, we can revisit then.

cmoulliard commented 4 years ago

Thank you for bringing your concerns to us. It seems like the discussion should be moved to the #cf-for-k8s slack channel in the cloudfoundry slack

Why. A github ticket, at least, allows to give access to the information discussed to everyone while Slack channel is only visible to the members. By doing that, you are killing in fact the discussion around helm adoption to install cf on k8s which is I think a mistake

Syerram commented 4 years ago

@cmoulliard We agree that this should be open for now and filed under the feature request category. We updated the title to reflect the problem and not a solution because helm may or may not be the right solution (for e.g. eirini is not going to ship helm anymore, istio is moving towards its own install mechanism).

We are focused on achieving pragmatic parity with cf features at the moment. We periodically review feature requests and assess them against the current priorities. We will update this ticket to indicate the appropriate status.

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/173736535

The labels on this github issue will be updated when the story is started.

jamespollard8 commented 4 years ago

Thanks everyone for the healthy conversation - as a maintainer I love seeing so many non-maintainers chiming in!

cmoulliard commented 4 years ago

We agree that this should be open for now and filed under the feature request category. We updated the title to reflect the problem and not a solution because helm may or may not be the right solution (for e.g. eirini is not going to ship helm anymore, istio is moving towards its own install mechanism).

I reviewed the description of the ticket ;-)

jbeda commented 3 years ago

Commenting from the outside. I'm not a super active member of the community here. But I was responsible for helping to standardize on ytt/kapp on the VMware side (and continuing to invest in those tools over time). I think that had a great influence on the choice here as part of this project.

While Helm has wide adoption, I personally am not a fan of the way that deployment and templating are tightly coupled. And, as a former member of the CNCF TOC, projects that are incubating or graduated are not the solution but rather a solution that has met a bar wrt security, maturity, governance and adoption. The CNCF is totally ok with competing projects and publicly aims not to be a "kingmaker".

On the technical issues here -- there is no silver bullet here.

The fact that ytt and kapp are two separate tools really enables people to integrate this with other workflows. For instance, the monolithic nature of helm creates problems with using it with gitops as having separate render and apply steps is much more natural (and fits with other tools in the space like kustomize or jsonnet or cue). helm template often has issues for more advanced helm charts. (Sidebar: I was talking to one of the main authors of the GitOps toolkit - https://github.com/fluxcd/toolkit - and he mentioned that they had to special case Helm for these reasons.)

Beyond that, the go template engine that helm uses is uniquely unsuited for templating YAML. Text based templatization is super error prone with a whitespace sensitive document. You can see this in something like the nginx chart.

IMO the ytt strikes a good balance between something that is structure aware, very portable without custom binary extensions and approachable in terms of using a variant of a popular language (starlark/python). Again, there is no silver bullet in this space and there will always be pros and cons to any solution.

Talking to the ytt team, they are taking the feedback that you gave above seriously and would like to smooth over those rough edges. I encourage you to file some issues over at the ytt repo. The team would love to engage on those specifics.

From my point of view -- I think that providing a fully rendered set of yaml with good defaults is a great thing for a project to have and I'd encourage that. It makes it easy for folks to get started without having to use a common toolset and is something I personally love to see in projects.

Finally, I trust that the folks working on cf-for-k8s have users at the top of their minds and are weighing off different options here as decisions like this are made. At the end of the day, in open source, technical decisions, when reasonable people can disagree, are made by those on the ground doing the work.

cmoulliard commented 3 years ago

Disclaimer 2: I am sceptical about the usage of a tool that is open source, but lacks widespread usage in the broader Kubernetes community. We had (well, have for current mainstream CF) such a tool: BOSH. It is really great in what it does, but it lacks adoption outside the CF community. I do see a risk that something similar will happen to the k14s tool suite. I wouldn't go as far as saying that the fate is decided yet though.

I completely agree with you and your disclaimer as currently Cloudfoundry is imposing to the developers to install many tools : 

brew tap vmware-tanzu/carvel
brew install ytt kbld kapp imgpkg kwt vendir yq
brew install helm
brew install cloudfoundry/tap/bosh-cli

to finally deploy a collection of YAML resources generated that we could easily install using helm tool only.

cmoulliard commented 3 years ago

IMO the ytt strikes a good balance between something that is structure aware, very portable without custom binary extensions and approachable in terms of using a variant of a popular language (starlark/python). Again, there is no silver bullet in this space and there will always be pros and cons to any solution.

Using ytt as tool is quite difficult to be honest and specifically when you have to merge content in a YAML template or extend an object which is part of a list/array.

I created such a template file 2 weeks ago and cannot remember why it has been finally designed like this

#@overlay/match by=overlay.subset({"kind":"Deployment"})
#@overlay/match by=overlay.subset({"metadata":{"name":"kpack-controller"}})
---
spec:
  template:
    spec:
      #@overlay/match missing_ok=True
      volumes:
        - name: custom-certs
          secret:
            secretName: cert-key
      containers:
        #@overlay/match by=overlay.subset({"name": "controller"})
        -
          #@overlay/match missing_ok=True
          volumeMounts:
            - name: custom-certs
              mountPath: /certs
          env:
            #@overlay/append
            - name: SSL_CERT_DIR
              value: /certs

Nevertheless this is also something which is not so easy to be achieved too using Helm ;-)

cmoulliard commented 3 years ago

From my point of view -- I think that providing a fully rendered set of yaml with good defaults is a great thing for a project to have and I'd encourage that. It makes it easy for folks to get started without having to use a common toolset and is something I personally love to see in projects.

+100. If CloudFoundry would like to promote this project and target specifically the developers, then this is needed as it will allow to install everything which is needed on a K8S cluster with 2 tools: helm + kubectl or even kubectl (if the static files are generated).

Ideally, the project to be installed should include:

I created a script for that purpose which should be extended to deploy stratos, the container registry,... : https://github.com/halkyonio/r-d/blob/master/cloudfoundry/scripts/install_cf_4_k8s.sh

bkrannich commented 3 years ago

Thanks for commenting again, @cmoulliard!

Recently, I was wondering if the two options are mutually exclusive: I believe that indeed bigger providers will need very sophisticated means to operate their (potentially big number of) cf-for-k8s deployments.

At the same time, I completely agree to the observation that the Carvel toolchain can have a similar role than BOSH had in the cf-on-vms world when it comes to creating a high entry barrier for people that are new to CF but maybe not so new to the K8s ecosystem.

To cater for both cases, my thought was: Can the individual components of cf-for-k8s define an API which is based on the configuration they support (ultimately, the filled out yaml including possible values) as opposed to making ytt templates the de-facto API. This approach would still allow vendors to use their toolchain of choice to come to the specific configuration but would allow a "plain yaml approach" for people who prefer kubectl apply -f and creating own helm charts for people who prefer to use this tooling.

Not sure what people think about the topic, but wanted to share my thinking here to receive feedback.

cmoulliard commented 3 years ago

Recently, I was wondering if the two options are mutually exclusive: I believe that indeed bigger providers will need very sophisticated means to operate their (potentially big number of) cf-for-k8s deployments.

They are not exclusive as it is needed that we support 2 profiles:

cmoulliard commented 3 years ago

At the same time, I completely agree to the observation that the Carvel toolchain can have a similar role than BOSH had in the cf-on-vms world when it comes to creating a high entry barrier for people that are new to CF but maybe not so new to the K8s ecosystem.

Why dont you then propose a container image packaging all the needed tools (bosh, ytt, kapp, kp, kubectl, ....) and document how it could be used locally ?

bkrannich commented 3 years ago

Why dont you then propose a container image packaging all the needed tools (bosh, ytt, kapp, kp, kubectl, ....) and document how it could be used locally ?

While this would avoid people having to install all the tooling locally, it would still be this "magic" step that takes a bunch of templates and turns it into something that's suddenly running on a cluster (if you include the kapp deploy step into what the image offers).

cmoulliard commented 3 years ago

To cater for both cases, my thought was: Can the individual components of cf-for-k8s define an API which is based on the configuration they support (ultimately, the filled out yaml including possible values) as opposed to making ytt templates the de-facto API. This approach would still allow vendors to use their toolchain of choice to come to the specific configuration but would allow a "plain yaml approach" for people who prefer kubectl apply -f and creating own helm charts for people who prefer to use this tooling.

This aspect could be handled by a Kubernetes controller or Operator able to generate on the cluster the objects/resources and consuming the data of the user using a CRD and containing by example, the credentials of the container registry, user/pwd to be used to access the CF API, ... Such a CRD could also be used to enable/disable the installation of some features such a private container registry, stratos console, ...

cmoulliard commented 3 years ago

it would still be this "magic" step

This is the reason why I think that the best approach then will be for the developers to propose something like this: Extend the existing "cf install/uninstall" client tool in order to install an operator or controller on the kubernetes cluster able to perform the magic stuff (see my comment around controller/operator and CRD) WDYT ?

bkrannich commented 3 years ago

Not sure. Wouldn't this mean that the developer API is another, smaller set of YAML configuration that then gets processed by the controller, translated into a broader set of YAML configuration "in cluster" that then gets picked up by the actual components?

So, a trade-off between

When I say "magic", what I mean are of course things like:

Thinking back to @julz original reasoning for proposing a plain YAML API for Eirini, I believe one of the arguments was "ability to use components separately from the entire cf-for-k8s".

cmoulliard commented 3 years ago

Not sure. Wouldn't this mean that the developer API is another, smaller set of YAML configuration that then gets processed by the controller, translated into a broader set of YAML configuration "in cluster" that then gets picked up by the actual components?

If we would like to avoid that we manage everything as the client side or let's say on the machine of the developer interested to provision a k8s cluster with cf-for-k8s which ultimately is installing different infrastructure components such as Eirini, Istio, CF API, kpack, ... then we should rely on a Server side component which is maybe a controller, operator (or maybe using a more elaborated tool such as https://crossplane.io/).

A new API is not needed as by example the CF API pod can manage the CRDs. Of course, this is like a snake eating its tail as it will needed that we have a YAML file able to first provision the CF API pod and infrastructure operator able to manage the installation of the cf-for-k8s components

cmoulliard commented 3 years ago

When I say "magic", what I mean are of course things like:

  • In the resulting YAMLs, there's multiple places where you have to put parameter X to make things consistent and the templating allows you to specify X only once
  • There's some "calculated values" and instead of specifying how you come to these calculated values you just take the inputs to the calculation as parameters
  • and so on

Such a find/replace, calculations, ... actions could also take place on the server side using a controller, pod, ... if you code such logic using go language or java (as now we can also create beautiful java operator - https://github.com/java-operator-sdk/java-operator-sdk). No need in this case to struggle with YAML templates, ... but instead to use Java classes (or go types if you prefer to code in go)

cmoulliard commented 3 years ago

My 5cents idea of the day ;-) to at least improve the situation

Why dont you package the needed cf/kubernetes tools such as: ytt, kapp, kpack, bosh, cf, ... within a container image as that already exists by example for helm, kubectl, ...: https://github.com/dtzar/helm-kubectl ?