Add support to refer K8S secrets with skipper filter

[tamer-abdulghani]

Backgroud We are trying to deploy ingress objects that uses Skipper. Those ingresses have Microsoft OAuth configuration which need a service principle (clientId/clientSecret). Currently we are storing these secrets in GitHub secrets and override the ingress objects during helm deployment to K8S. To avoid storing these client-id/client-secret in GitHub, we are thinking to store them directly in K8S secrets. However, then we must need to have a way to refer those secrets in ingresses objects which is currently not available in Skipper afaik.

Proposal Add support to refer K8S secrets in ingresses objects that uses skipper ingress class.

My current ingress object looks like: as you see, I'm overriding client-id/client-secret during helm deployment. But it would be much better and more secure to refer them somehow to K8S secrets.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "{{ .Chart.Name }}-s-ingress"
  annotations:
    kubernetes.io/ingress.class: skipper
    zalando.org/skipper-filter: | 
      oauthOidcUserInfo(https://login.microsoftonline.com/TENAND_ID/v2.0, 
          "{{ .Values.ingress.clientId }}", 
          "{{ .Values.ingress.clientSecret }}", 
          https://{{ .Values.ingress.host }}/oauth/callback, 
          "email profile", 
          "",
          "", 
          "X-Remote-User:claims.email",
          "0"
          )
          -> oidcClaimsQuery("/:{{- include "groupFilter"  . -}}",
            "/:@_:email%\[*@mydomain.com\](mailto:*@mydomain.com\)")
          -> dropRequestHeader("Cookie")
  labels:
    app: "{{ .Release.Name }}"
spec:
  rules:
    - host: "{{ .Values.ingress.host }}"
      http:
        paths:
        - path: "/{{  .Values.ingress.path }}/*"
          pathType: ImplementationSpecific
          backend:
            service:
              name: "{{ .Chart.Name }}-svc"
              port:
                number: 8080

[szuecs]

@tamer-abdulghani , I would not do it via Kubernetes secrets API, but use the secrets module that works with the filesystem. Kubernetes secrets can be mounted to the pod, so you can configure skipper to update these in memory secrets. If you want to propose a filter change it would be great to see.

[AlexanderYastrebov]

Some thoughts for the context.

We could establish some kind of convention to lookup value from the secrets.SecretsReader e.g. if client_id/client_secret starts with a prefix like secret:/opt/ms-client-id. We would also need to consider multitenant setups such that users that define ingress can not read client id/secret not intended for them.

Currently we use secrets.SecretsReader for https://github.com/zalando/skipper/blob/master/docs/reference/filters.md#bearerinjector

The https://github.com/zalando/skipper/blob/master/docs/reference/filters.md#oauthgrant also uses secrets reader configured separately https://github.com/zalando/skipper/blob/master/docs/tutorials/auth.md#configure-oauth2-credentials

[spr-mweber3]

Any news on this?

Just stumbling across the same problem. Really need to have the option to have the secrets either put into the filesystem or Kubernetes secrets. I understood that oauthGrant already has this.

[sanjeev55]

Hi, I am trying to use the "oauthOidcUserInfo" filter, but have stumbled upon the following error.

failed to create filter \"oauthOidcUserInfo\": failed to read secrets from secret source: open : no such file or directory"

I am not aware of any secret source open. What could have been the issue? I am sorry but I am really new to this, so if any of you can help, then that would be great.

[szuecs]

@sanjeev55 please create a separate issue and add more context. For example what do you try to achieve and how do you run skipper.

[fragsalat]

We also had the use case where we want to add an access token to the proxied request. The access token should be stored and read using a k8s secret.

Example route

    - pathSubtree: /backend/3rd-party-api
      filters:
        - preserveHost("false")
        - setPath("/api/", "")
        - setRequestHeader("api-token", "VALUE_FROM_SECRET_INSTEAD_PLAINTEXT")
      backends:
        - backendName: 3rdPartyApi

[szuecs]

@fragsalat how do you suppose to reference the secret? I guess you would need to reference by namespace and name, but then the filter would be bound to kubernetes only, which is not what skipper project really wants to provide.

[fragsalat]

You are right. I thought about reading it from a file on a mounted volume but a route group can not have a mounted volume :(

When having a mounted volume I think skipper already has a similar feature. The problem with that is, the bearer injector only puts it into the authorization header. If there would be a function to read from files. Something like - setRequestHeader("api-token", readFromFile("/tmp/secrets/3rd-party-token"))

But you are right. With a routegroup this would not help.

[szuecs]

@fragsalat the biggest challenge I see is that developers who create ingress/routegroup resources are not the ones that run skipper-ingress. The ones that runs skipper-ingress can mount secrets into it, the others can not. It would be some breach in multi-tenant environment to allow this. I don't mind having a solution for this, if there is a good idea how to do it, but for the zalando case, I am confident that this won't be possible to allow. We can discuss it in chat internally.

[AlexanderYastrebov]

@fragsalat

We also had the use case where we want to add an access token to the proxied request. The access token should be stored and read using a k8s secret.

If there would be a function to read from files. Something like setRequestHeader("api-token", readFromFile("/tmp/secrets/3rd-party-token"))

You can deploy skipper as a standalone deployment proxy and configure it to inject token from mounted secret like so:

apiVersion: apps/v1
kind: Deployment
metadata:
...
spec:
...
    spec:
      volumes:
        - name: credentials
          secret:
            secretName: my-secret
      containers:
        - name: proxy
          image: registry.opensource.zalan.do/teapot/skipper:latest
          volumeMounts:
            - name: my-secret
              mountPath: /tokens
              readOnly: true
          args:
            - skipper
            - -address=:9090
            - -wait-for-healthcheck-interval=20s
            - -credentials-paths=/tokens
            - -credentials-update-interval=1m
            - -inline-routes
            - |
              main: *
                -> bearerinjector("/tokens/my-token")
                -> modRequestHeader("Authorization", "^Bearer (.+)$", "$1")
                -> copyRequestHeader("Authorization", "api-token")
                -> dropRequestHeader("Authorization")
                -> preserveHost("false")
                -> "https://foo.test";
              health_up: Path("/healthz") -> inlineContent("OK") -> <shunt>;
              health_down: Path("/healthz") && Shutdown() -> status(503) -> inlineContent("shutdown") -> <shunt>;
...

It has to modify and rename Authorization header added by bearerinjector filter.

Maybe we can create a generic filter setRequestHeaderFromSecret("<header name>", "<secret name>", "<optional secret value prefix>").

This case then would be simply setRequestHeaderFromSecret("api-token", "/tokens/my-token") and bearerinjector("/tokens/my-token") would be equivalent to setRequestHeaderFromSecret("Authorization", "/tokens/my-token", "Bearer ")

If secret is static then you can just put it to the env variable and use k8s variable env substitution:

main: *
  -> setRequestHeader("api-token", "$(API_TOKEN)")
  -> preserveHost("false")
  -> "https://foo.test";

See

• https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-container-environment-variables-using-secret-data
• https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#use-environment-variables-to-define-arguments

[jbuettnerbild]

setRequestHeaderFromSecret is a better solution. Mount the Kubernetes Secret, set credentials-paths and you can now set credentials as a header.

I use it for access of a api gw protected by a api key.

example:

   ...
    volumeMounts:
      - name: secrets
        mountPath: /etc/skipper/secrets
        readOnly: true
    volumes:
      - name: secrets
        secret:
          secretName: "skipper-secrets"
   ...

skipper-config.yaml: |-
          application-log-level: DEBUG
          address: ":9999"
          credentials-update-interval: 60m
          credentials-paths: ["/etc/skipper/secrets"]
         ...

         inline-routes: |-
            test: PathRegexp("^/test")
              -> setRequestHeaderFromSecret("x-api-key", "/etc/skipper/secrets/api-key")
              -> preserveHost("false")
              -> modPath("/test", "")
              -> "https://api-gw.example.com";

https://opensource.zalando.com/skipper/reference/filters/#setrequestheaderfromsecret