Confusing documentation about istio operator decommission

cfBrianMiller commented 2 years ago

Summary of issue

Documentation for the operator states the following

Use of the operator for new Istio installations is discouraged in favor of the Istioctl and Helm installation methods. While the operator will continue to be supported, new feature requests will not be prioritized.

However the alternatives proposed are confusing

Helm

The proposed installation method using Helm page specifically states

This feature is currently considered alpha.

Alpha features not preferred for production deployments.

Istioctl

The istioctl option points back to the operator CRD as the recommended method for production installations.

Providing the full configuration in an IstioOperator CR is considered an Istio best practice for production environments. It also gives you the option of completely delegating the job of install management to an Istio Operator, instead of doing it manually using istioctl.

Requested Resolution

Can you please provide

Clear documentation for the preferred installation / upgrade method
Reason the istio operator is being deprecated

The only reference I was able to find on the istio operator deprecation was #10420 which is unclear as to the reasoning.

howardjohn commented 2 years ago

The istioctl doc is not saying to use the Istio operator, just to pass a file instead of --set flags. The input to this is - confusingly - IstioOperator

Kampe commented 2 years ago

Istio operator is extraordinarily useful to lay down for any gitops flavor deployments via kustomize or helm. @howardjohn what would you recommend we move towards using?

howardjohn commented 2 years ago

Whatever you use for your own apps - I imagine you don't have an operator for each of them. If you use Helm, then install with Helm. If you use other tools that work with plain k8s yaml, then render an Istio install (helm template or istioctl manifest generate) and use that.

GitOps is not a well defined thing, but IMO GitOps != operators, and I would go as far as to say that most operators are in conflict with best practices for GitOps. That is just moving your config from being well defined in your git repo to being defined to being defined in 50k LOC in some foreign git repo (http://github.com/istio/istio). See https://www.solo.io/blog/configuration-as-data-gitops-and-controllers-its-not-simple-for-multi-cluster/ for more discussion

Kampe commented 2 years ago

Whatever you use for your own apps - I imagine you don't have an operator for each of them. If you use Helm, then install with Helm. If you use other tools that work with plain k8s yaml, then render an Istio install (helm template or istioctl manifest generate) and use that.

GitOps is not a well defined thing, but IMO GitOps != operators, and I would go as far as to say that most operators are in conflict with best practices for GitOps. That is just moving your config from being well defined in your git repo to being defined to being defined in 50k LOC in some foreign git repo (http://github.com/istio/istio). See https://www.solo.io/blog/configuration-as-data-gitops-and-controllers-its-not-simple-for-multi-cluster/ for more discussion

Would love that option, however is kustomize not supported?

If we had raw manifests we could point at this would be mostly a non issue as well.

How we currently solve this with kustomize

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - https://raw.githubusercontent.com/istio/istio/1.11.3/manifests/charts/base/crds/crd-operator.yaml
  - https://raw.githubusercontent.com/istio/istio/1.11.3/manifests/charts/base/crds/crd-all.gen.yaml
  - https://raw.githubusercontent.com/istio/istio/1.11.3/manifests/charts/istio-operator/files/gen-operator.yaml
  - https://raw.githubusercontent.com/istio/istio/1.11.3/samples/addons/kiali.yaml
  - istio-profile.yaml

Don't particularly care if this is managed via an operator or not, but the ability to declaratively lay down an istio install is wildly useful, and dare I say, far more ideal then downloading (and updating) Yet Another CLI to essentially do the same thing?

howardjohn commented 2 years ago

I would suggest not doing that as our github repos pull from gcr.io/istio-testing which are not production verified images. We don't currently have a way to just point to a URL for kustomize

Kampe commented 2 years ago

I would suggest not doing that as our github repos pull from gcr.io/istio-testing which are not production verified images. We don't currently have a way to just point to a URL for kustomize

Gotcha, temperature check but do you see the appeal of what I've described here? This seems like a simple feature (addition of kustomization.yaml) add that would in theory greatly simplify installs.

howardjohn commented 2 years ago

Yes, but most people do that with Helm using things like helmfile, terraform, etc.

BTW one of our priorities for the next release is to publish reference gitops setups

howardjohn commented 2 years ago

I have nothing against kustomise btw, we mostly didn't do it because we don't have a good way to publish to a GitHub branch and they don't have a good way to pull from anywhere but git (last I checked)

Kampe commented 2 years ago

Yes, but most people do that with Helm using things like helmfile, terraform, etc.

BTW one of our priorities for the next release is to publish reference gitops setups

Excellent that would help a ton to have a defined path, we'll move to using helm install 👍

I have nothing against kustomise btw, we mostly didn't do it because we don't have a good way to publish to a GitHub branch and they don't have a good way to pull from anywhere but git (last I checked)

We love kustomize, and it's built into kubectl(-ish 😏 ) - in theory you can point directly at any yaml file like we're doing above ^^

johnkeates commented 2 years ago

Hi there, I arrived here after a similar path when we were looking into upgrading our Istio setup (currently using the operator), and was a bit confused about what the preferred method is at this time.

While others might not arrive here the the same way (read the current docs, find out Operator is discouraged, try to find out why, go to the commit that put the message in the FAQ, find issue #10420, then find this issue), I do think that some help along the way to find that TOC discussion would greatly benefit everyone. The closest meeting I could find was https://youtu.be/aARpLyLGbw8?t=7 but at that stage the Operator Install was put on the 'future' agenda. That said, it's still great that we have plenty of meeting recordings giving a good idea of the design and discussion process around various Istio WGs.

Edit: for future reference, found it here: https://youtu.be/DhSWiXodREc?t=477 since it was a TOC meeting and not an Environments meeting after all. Makes sense to have the in-cluster operator less prevalent if maintenance (of the code) and risk (of having a highly privileged operator) are just not in a happy place.

cdm-arm commented 1 year ago

Hi together, while I love the pointing to the yt videos by @johnkeates and and the discussion above it still leaves people reading the docs at an unsatisfied stage:

Use IstioOperator - discouraged
Use helm - alpha, so not production ready
Use istioctl - "for in-depth evaluation or production use"
- above recommended manifest generation seems also not production ready due to a lot of "following caveats" listed.
- using istioctl directly isn't compatible with gitops deployments.

So anyone who has deeper insides would be welcomed to give a proper recommendation on how to get istio gitops-style like into a cluster.

johnkeates commented 1 year ago

We have started to do a mix:

Helm where we can validate ahead of time
Plain manifests (generated by istioctl, edited by us) everywhere else

I suspect a lot of this intermediate state comes from the merger of most things into istiod, and the somewhat simpler deployment of everything where you almost don't need a chart or operator to use it. Besides CRDs and a few deployments, you only need to do things that you have to think about for your specific case anyway like what ingress setup you're going to use.

I do see how this confused state came to be; people that use it already 'know' how to get their setup the way they like it, and people who are new might not even know what it is that they are here for. That 'gap' makes it so that it can become hard for 'knowing' people to write basic instructions for people 'who do not know'.

A similar issue existed in Rancher Desktop and traffic for example where people who knew Lima, Rancher Desktop Agent, Flannel, VDE and Klipper-LB can get started right away, but if not, you'd have a hard time finding out how to get traffic in without using port forwarding. They solved that by creating a simple "how to setup nginx ingress instead of traefik" which gets you to a level where you know that ingress, svc and LB configuration works out of the box and is actually not that different from a random Kubernetes cluster. That knowledge ports over to any other traffic things you might have wanted so even new users can hit the ground running.

The equivalent for Istio might be a bit harder (So far, I can't come up with anything either), but I'd say that like the 'tasks' section, there could be a better 'setup' section.

It might also be much simpler than this whole story I just wrote: a few releases later the alpha label gets removed and we're all fine again; until then, use the operator.

howardjohn commented 1 year ago

Where did the conclusion "use the operator" come from? It is the only one that has liberal warning signs all over to NOT use it. If these warnings are not sufficient maybe we should consider removing the doc from the main "setup" page to avoid confusion.

johnkeates commented 1 year ago

Helm and Istioctl have never been production ready, so that only leaves the operator, no matter how discouraged. (or the other way: manifests, but that's not really friendly to newcomers)

howardjohn commented 1 year ago

Use istioctl - "for in-depth evaluation or production use"

https://istio.io/latest/about/faq/#setup also answers this.

Helm is like 99.9% of the way to being promoted, btw, so its reasonable to use at this point.

Since there is still so much confusion I think I am going to push to get all docs about the operator moved to somewhere that makes it very clear its not the recommended approach.

johnkeates commented 1 year ago

I see, I suppose I was still looking at it from a GitOps perspective. If one were to use not-GitOps (HumanOps?) that would totally be production-usable.

cdm-arm commented 1 year ago

Where did the conclusion "use the operator" come from? It is the only one that has liberal warning signs all over to NOT use it. If these warnings are not sufficient maybe we should consider removing the doc from the main "setup" page to avoid confusion.

Would be a small but fruitful change. Thanks for confirming helm's status, makes the decision to use them for production too easier.

howardjohn commented 1 year ago

I see, I suppose I was still looking at it from a GitOps perspective. If one were to use not-GitOps (HumanOps?) that would totally be production-usable.

https://blog.howardjohn.info/posts/operators/index.html#gitops for my take

istio / istio.io