search

3scale-ops / marin3r

Lightweight, CRD based envoy control plane for kubernetes
Apache License 2.0
56 stars 12 forks source link

Ability to load secrets other than of type `kubernetes.io/tls`? #218

Open GreenCappuccino opened 5 months ago

GreenCappuccino commented 5 months ago

When using envoy.extensions.filters.http.oauth2.v3.OAuth2, credentials.token_secret and credentials.hmac secret are required, which don't fit into TLS Secrets. From what I can tell marin3r doesn't load non-TLS secrets, and I cannot specify extraVolumeMounts in EnvoyDeployment to work around this issue.

GreenCappuccino commented 5 months ago

/kind feature

roivaz commented 5 months ago

Hi @GreenCappuccino! Thanks for the feedback!

As you mention, an extraVolumeMounts field in the EnvoyDeployment spec would workaround this. The downside with path based sds is that updates are not picked up immediately. The kubelet has a refresh interval (tipically configured to 60s) for refreshing secrets and configmap changes in pods, so if you require those secrets to be immediately refreshed by envoy when changed, that solution won't do.

The other option is to expose in the API something like this:

    - type: secret
      generateFromSecret: certificate
      blueprint: genericSecret

This would allow secrets of type Opaque to be used to produce an envoy generic secret proto. I'm assuming that a generic secret is what's required for your use case, let me know if I'm wrong.

I'd rather go for the second option as it has the advantage that changes to the secrets are propagated immediately and that works both for EnvoyDeployment and for sidecars.

wdyt?

GreenCappuccino commented 5 months ago

Hello, thank you for responding

I'm assuming that a generic secret is what's required for your use case, let me know if I'm wrong.

Interestingly the OAuth2 Credentials take SdsSecretConfig, I think that that is a different resource? I think the ads and api_config_source fields of the ConfigSource on that should be of interest.

EDIT: I think I'm mistaken, if only name is specified then I think it can reference a static generic secret?

I think the most ideal solution would be to be able to create such secrets as you described in the second option, though I'm not entirely sure of what that entails for implementation.

extraVolumeMounts (also for the sidecar?) and extraVolumes could also potentially be useful as a stopgap and be useful beyond the scope of this issue, I'm not sure. But yes, it would just a workaround from not having a native secret solution in this case.

roivaz commented 4 months ago

Hi @GreenCappuccino, sorry for taking so long to reply. I've implemented the possibility of loading k8s opaque secrets into envoy generic secret protos (see https://github.com/3scale-ops/marin3r/pull/221). I have tested this with the oauth2 filter and seems to work fine. You can check the example in https://github.com/3scale-ops/marin3r/tree/feat/generic-secret/examples/k8s/oauth2. If interested in trying it yourself, you can use the image quay.io/3scale/marin3r:v0.13.0-alpha.2, which contains the changes.