[Feature] Allow calling in cluster services

[samj1912]

Problem Statement

There is currently no easy way to extend kyverno capabilities. Allowing kyverno to make calls to in-cluster services could unlock such capabilities.

Solution Description

Allow a new serviceCall context variable type.

rules:
- name: example-service-call
  context:
  # name of the variable to capture the service output from
  - name: output
    serviceCall:
      service:
        name: service-name
        namespace: some-namespace
        # optional
        port: 80
        # optional
        path: validate-namespace
        # optional url GET params
        params:
          namespace: '{{ request.object.metadata.namespace }}'
        # optional JSON post data. If defined.
        data:
          method: '{{ request.method }}'
        # http verb for the call. One of the http verb names. Defaults to GET if data is not defined otherwise POST
        verb: POST
      # optional timeout in seconds per retry, default is 2s
      timeout: 10
      # optional retries. Defaults to 1
      retries: 2
      # optional jmespath expression evaluated on the response before the variable is assigned 
      jmesPath: "items | length(@)"

The above service calls would limit the return accept type to json only. The encoding would also be json for the post data.

We would require an in cluster Service object to be defined to reach out to.

Alternatives

No response

Additional Context

No response

Slack discussion

No response

Research

• [X] I have read and followed the documentation AND the troubleshooting guide.
• [X] I have searched other issues in this repository and mine is not recorded.

[samj1912]

cc: @chipzoller, @JimBugwadia, @realshuting This would allow easy extensions for kyverno to be created which can then be used for validation, mutation or generation.

Related https://open-policy-agent.github.io/gatekeeper/website/docs/externaldata#providers

[samj1912]

We can target this for 1.8

[chipzoller]

Agree, this would be valuable.

[JimBugwadia]

Certainly possible to do and has some interesting use cases!

While an API call introduces flexibility for data lookups, etc. I am not sure how it can be used as an extension. I was envisioning an extension of more of an SDK plugin vs an API call.

For either approach, the security and performance considerations would be important to work through in detail. Can we capture in a KDP?

[chipzoller]

One of the main questions I'd have around using this is handling authentication to those other services. As soon as we start going down that path, this becomes a very slippery slope.

[chipzoller]

Do you want to covert this to a KDP? Would you plan on developing this for 1.8.0 potentially?

[Dentrax]

Dropping some use case about the proposal as answering of @JimBugwadia's comment: https://github.com/kyverno/kyverno/pull/3820#issuecomment-1191710517

@Dentrax - are you looking for this to address a use case? If so, I would like to learn more about that.

Image you are running hundreds of clusters and want to create some excluding policies using ConfigMap. For example, we want to exclude some critical images such as istio, node-local-dns etc. But this task requires some stuff such creating ConfigMaps in every cluster, and we have to keep maintaining those.

I'd like to store my context in a centralized config solution such as consul. By doing so, we are able to store all excluding rules in a single place for better maintainability and real-time operation for upserting the new configs.

But I just noticed this issue is for in-cluster services only. What I was looking for is that external data. (I actually mean out-cluster arbitrary apiCall) Should we create another KDP for this?

rules:
- name: get-excluded-images
  context:
  - name: excludedImages
    apiCall:
      urlPath: "https://consul.test.com:8500/v1/kv/exluded-images"

cc @developer-guy

[JimBugwadia]

@Dentrax - thanks for the example use case!

In-cluster, or an external API, should work the same. Both will need to be secured.

So, going along with your Consul example, how would you specify the encryption and authentication / authorization for the Consul API?

• Encryption: provide a certificate / root CA?
• Authentication: username and password? token? cert? something else?

In either case, the network policy would also need to be configured to allow this call.

[zswanson]

I'd be interested in leveraging this with an internal 'metadata' service we have that our application devs populate about each app during application onboarding. This contains a lot traceability data such as team ownership, cost allocation data, etc. We have a desire to allow/deny certain kubernetes attributes based on that metadata, such as allowing an Ingress to declare a 'high priority' ingressClassName only if its owned by certain groups (and actually validating that ownership through the service). The metadata service is hosted within a central kubernetes cluster so we'd need both in-cluster and external-cluster calls to it.

[chipzoller]

@zswanson how would you expect to handle authN/authZ to these external services? Do you have requirements there?

[zswanson]

@chipzoller Looks like this particular internal service doesn't require any auth for GET right now. The rest of the service (which we wouldn't need to use) are Bearer token JWT. I think generally our in-house stuff uses username/password or JWT tokens.

[eddycharly]

Isn't it going to be tricky to implement in CLI when trying to validate manifests against rules without deploying ?

[aslafy-z]

@eddycharly afaik, mocking of the resulting context values exists in the test framework. There's no mock of the http request in itself.

[chipzoller]

Why don't we start simple with this feature (and put it behind an alpha feature flag) by getting the basics down such as:

• HTTP only
• No custom headers
• GET method
• JSON response only

[chipzoller]

Closing this issue since the KDP was approved and a PR has already been merged, tracking for release in Kyverno 1.10.