search

cloudfoundry / cf-k8s-networking

building a cloud foundry without gorouter....
Apache License 2.0
32 stars 17 forks source link

Operator can manage apps domains and their TLS certs without modifying "system" resources #32

Closed rosenhouse closed 4 years ago

rosenhouse commented 4 years ago

Looking at the workflow here I'm realizing that it works, but isn't optimal. The operator has to modify a gateway in the istio-system namespace, which was laid-down by cf-for-k8s itself.

A better workflow would be that the operator creates and manages their own Gateway resources, possibly in cf-system or cf-workloads namespaces. They don't need to change anything in the istio-system namespace. That is easier for them to manage via declarative config + GitOps. It is also safer for upgrades: their changes are at less risk of being blown away by a redeploy of cf-for-k8s.

Thoughts?

tcdowney commented 4 years ago

@rosenhouse are you referring to creating the secrets in the istio-system namespace as being something they shouldn't be doing? I believe that's the only interaction with istio-system we have since it is referencing secrets that the istio-ingressgateway pod in that namespace will use. I think https://github.com/istio/istio/issues/14598 needs to happen if we want the secrets to live in other namespaces.

The workflow you referenced talks about modifying a Gateway resource that lives in cf-workloads or cf-system. Maybe it's the instructions themselves in that doc which are confusing?

  1. Add new server in the edge Istio gateway object in cf-workloads, e.g.

I can see how that could be worded better.

rosenhouse commented 4 years ago

Ah, I was confused by the location of the secret: https://github.com/cloudfoundry/cf-k8s-networking/pull/28/files#r388619809

ndhanushkodi commented 4 years ago

We now have one gateway object that we want to be the source of truth, the one in cf-system here

Using one gateway allows us to support app and system domains being the same, without having conflicting domains in two gateways. Because the gateway has to manage both app and system domains, we think it makes sense that it belongs in cf-system. What do you think @rosenhouse?

We also plan to update the referenced doc to reflect the single gateway.

That being said, we realized that there is one more gateway object created in istio-system which has:

- hosts:
  - '*'

and we're not sure if that's necessary. We're writing a story to see if we can not include the gateway in istio-system.

-- Nitya and @christianang

rosenhouse commented 4 years ago

Sounds good to me. I think it would be cleaner if we didn't create the istio-ingressgateway in istio-system. that's part of the default install, but we don't use it, so I'd remove it.

If other software on the cluster wants to piggyback on the cf-provided istio, they're breaking the rules, and should at the very least have to bring their own gateway.

ndhanushkodi commented 4 years ago

Great, we've captured this information in this story where we'll delete that gateway and document that decision.

Closing this issue :)