Configure Envoy Proxy fleet in different namespaces

envoyproxy / gateway

Manages Envoy Proxy as a Standalone or Kubernetes-based Application Gateway

https://gateway.envoyproxy.io

Apache License 2.0

1.64k stars 354 forks source link

Configure Envoy Proxy fleet in different namespaces #2629

Open arkodg opened 9 months ago

arkodg commented 9 months ago

          thinking out loud, potential API could look like

deploymentMode:
  watch:
    namespaces: .........
  deploy:
    envoy:
      namespaceType: GatewayNamespace

By default (when this field is unset), EG watches all resources in all namespaces, and depoys infra in the namespace where EG is running (ControllerNamespace)
watch.namespaces can be used to set the specific namespaces to watch resources (Gateway, xRoute, Service etc)
deploy.envoy.namespaceType can be used to specify to EG to deploy proxy infra to deploy envoy deployments in the namespace where the Gateway is created in. This option will need a helm knob too, to configure added permissions to be able to create deployments and services in other namespaces

Originally posted by @arkodg in https://github.com/envoyproxy/gateway/issues/1117#issuecomment-1492387115

cnvergence commented 5 months ago

I would like to tackle this one

cnvergence commented 5 months ago

It will also be necessary to move away from mTLS for control plane auth to sTLS + JWT Authn to validate the envoy. cc @arkodg

arkodg commented 5 months ago

yah @cnvergence, we can selectively do it for the case when the Gateway is running in a user namespace

github-actions[bot] commented 4 months ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

github-actions[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

kamalmarhubi commented 1 month ago

I believe this would be solved once GEP-1762: In Cluster Gateway Deployments is implemented, which is tracked in #4330 as referenced just above. Or I know at least the use case that led me to this issue would be solved by it.

arkodg commented 1 month ago

@kamalmarhubi can you share your use case ? we already support customizing gateway infrastructure https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/, the generated resources live in the envoy-gateway-system namespace, instead of the Gateway namespace

kamalmarhubi commented 1 month ago

@arkodg thanks yes, I'm already customizing it!

The reason I'd like the resources to be created in the Gateway namespace is that it allows cleaner RBAC policies. I'd like owners to be able to inspect their gateways' generated resources without necessarily giving them access to the gateway controller itself.

It also makes setting up network policies easier: with a default-deny setup, we need to add allow-egress policies in the envoy-gateway-system which seems messy. It also makes RBAC less clean: gateway owners either have to ask someone with appropriate permissions in envoy-gateway-system to do this for them, or we need to grant all gateway owners netpol management permissions in envoy-gateway-system. Neither of those options are ideal.

Do you expect that Envoy Gateway will implement GEP-1762?

arkodg commented 1 month ago

the inability to target the generated resources when creating a resource like network policy, is a fair point @kamalmarhubi Envoy Gateway already implements most of GEP-1762, we would need to implement this issue to completely implement the GEP

kamalmarhubi commented 1 month ago

~@arkodg while continuing to deploy envoy-gateway, I came across another reason to want to run envoy in the Gateway's namespace: cert-manager has an integration with the Gateway API and can create certificates for annotated Gateway objects.~

~However, the certificates it creates (and so the TLS secrets) live in the Gateway's namespace, and that's a limitation they're unlikely to attempt to lift because ownerReferences don't work across namespaces (https://github.com/cert-manager/cert-manager/issues/5610). Since workloads can't reference secrets from other namespaces, the envoy deployment or daemonset wouldn't be able to read those secrets.~

~It's easy enough to manually create the required certificates, but the cert-manager integration would be a handy shorthand, and would also make it harder to mess up by letting the certificate's DNS names and Gateway's TLS listeners diverge.~

edit: Never mind I think I was incorrect; this is documented on the Envoy gateway website.

cnvergence commented 1 month ago

Hey all, slowly returning back to speed, I should have some cycles to work again on EG Looking to move this one forward

arkodg commented 1 month ago

thanks @cnvergence, thinking out loud you should be able to use the k8s service account token and append it the request, may need changes to the xds cluster , and perform sTLS + JWT Validation in the xds server for this case / if token exists

github-actions[bot] commented 4 days ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days.