search

kubernetes-sigs / custom-metrics-apiserver

Framework for implementing custom metrics support for Kubernetes
Apache License 2.0
504 stars 176 forks source link

Serve OpenAPI spec #65

Closed johanneswuerbach closed 4 years ago

johanneswuerbach commented 4 years ago

I was looking into resolving https://github.com/DirectXMan12/k8s-prometheus-adapter/issues/257 and https://github.com/DirectXMan12/k8s-prometheus-adapter/issues/254 and it seems the best option would be to either serve and included an openapi spec here or provide an overridable cmd method.

Would that be correct @s-urbaniak ?

k8s-ci-robot commented 4 years ago

Welcome @johanneswuerbach!

It looks like this is your first PR to kubernetes-sigs/custom-metrics-apiserver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/custom-metrics-apiserver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. :smiley:

s-urbaniak commented 4 years ago

@johanneswuerbach thanks a lot for the contribution! cc'ing @lilic who wanted to have a look at this as well.

johanneswuerbach commented 4 years ago

Yes, log lines similar to:

I1104 23:44:18.242807       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.custom.metrics.k8s.io
E1104 23:44:18.243652       1 controller.go:114] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: OpenAPI spec does not exist

This is inspired from the way metrics server fixed the issue https://github.com/kubernetes-sigs/metrics-server/pull/65/commits/06bcca5d06eae0a46fd1909a733e9f8be85b74d1, but I'm not sure whether having a pre-build spec is the right approach here as this is more used like a library?

Additionally I guess those specs need to be customizable as https://github.com/DirectXMan12/k8s-prometheus-adapter needs to ship specs for resource, custom and external metrics.

paulfantom commented 4 years ago

This seems to work. However injecting it as is into prometheus-adapter creates an issue with lack of k8s.io/metrics/pkg/apis/metrics/v1beta1.NodeMetrics, so this alone won't be enough to fix https://github.com/DirectXMan12/k8s-prometheus-adapter/issues/257

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

renan commented 4 years ago

/remove-lifecycle stale

Any updates on this?

How can one help in order to move this forward?

johanneswuerbach commented 4 years ago

What is left is essentially to make spec itself providable by users of this library instead of hardcoding it here https://github.com/kubernetes-sigs/custom-metrics-apiserver/pull/65/files#r408868352

I'll try to find some time next week, but can't promise anything.

johanneswuerbach commented 4 years ago

@renan this should be reviewable now.

Instead of embedding an openapi spec in the custom-metrics-apiserver itself, the adapter can now provide a spec for all provided types and I extended to test-adapter with an example how this would look like.

Installing the test-adapter before this change into a kind cluster, resulted in no output for:

kubectl get --raw /openapi/v2 | jq . | grep 'custom.metrics'
kubectl get --raw /openapi/v2 | jq . | grep 'external.metrics'

Now I get:

kubectl get --raw /openapi/v2 | jq . | grep 'custom.metrics'
    "/apis/custom.metrics.k8s.io/": {
    "/apis/custom.metrics.k8s.io/v1beta1/": {
    "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace}/metrics/{name}": {
        "description": "list custom metrics describing an object or objects",
              "$ref": "#/definitions/io.k8s.metrics.pkg.apis.custom_metrics.v1beta1.MetricValueList"
    "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace}/{resource}/{name}/{subresource}": {
        "description": "list custom metrics describing an object or objects",
              "$ref": "#/definitions/io.k8s.metrics.pkg.apis.custom_metrics.v1beta1.MetricValueList"
    "/apis/custom.metrics.k8s.io/v1beta1/{resource}/{name}/{subresource}": {
        "description": "list custom metrics describing an object or objects",
              "$ref": "#/definitions/io.k8s.metrics.pkg.apis.custom_metrics.v1beta1.MetricValueList"
    "/apis/custom.metrics.k8s.io/v1beta2/": {
    "/apis/custom.metrics.k8s.io/v1beta2/namespaces/{namespace}/metrics/{name}": {
        "description": "list custom metrics describing an object or objects",
              "$ref": "#/definitions/io.k8s.metrics.pkg.apis.custom_metrics.v1beta2.MetricValueList"
    "/apis/custom.metrics.k8s.io/v1beta2/namespaces/{namespace}/{resource}/{name}/{subresource}": {
        "description": "list custom metrics describing an object or objects",
              "$ref": "#/definitions/io.k8s.metrics.pkg.apis.custom_metrics.v1beta2.MetricValueList"
    "/apis/custom.metrics.k8s.io/v1beta2/{resource}/{name}/{subresource}": {
        "description": "list custom metrics describing an object or objects",
              "$ref": "#/definitions/io.k8s.metrics.pkg.apis.custom_metrics.v1beta2.MetricValueList"
    "io.k8s.metrics.pkg.apis.custom_metrics.v1beta1.MetricValue": {
          "group": "custom.metrics.k8s.io",
    "io.k8s.metrics.pkg.apis.custom_metrics.v1beta1.MetricValueList": {
            "$ref": "#/definitions/io.k8s.metrics.pkg.apis.custom_metrics.v1beta1.MetricValue"
          "group": "custom.metrics.k8s.io",
    "io.k8s.metrics.pkg.apis.custom_metrics.v1beta2.MetricIdentifier": {
    "io.k8s.metrics.pkg.apis.custom_metrics.v1beta2.MetricValue": {
          "$ref": "#/definitions/io.k8s.metrics.pkg.apis.custom_metrics.v1beta2.MetricIdentifier"
          "group": "custom.metrics.k8s.io",
    "io.k8s.metrics.pkg.apis.custom_metrics.v1beta2.MetricValueList": {
            "$ref": "#/definitions/io.k8s.metrics.pkg.apis.custom_metrics.v1beta2.MetricValue"
          "group": "custom.metrics.k8s.io",
kubectl get --raw /openapi/v2 | jq . | grep 'external.metrics' 
    "/apis/external.metrics.k8s.io/": {
    "/apis/external.metrics.k8s.io/v1beta1/": {
    "/apis/external.metrics.k8s.io/v1beta1/namespaces/{namespace}/{resource}": {
        "description": "list external metrics",
              "$ref": "#/definitions/io.k8s.metrics.pkg.apis.external_metrics.v1beta1.ExternalMetricValueList"
    "io.k8s.metrics.pkg.apis.external_metrics.v1beta1.ExternalMetricValue": {
          "group": "external.metrics.k8s.io",
    "io.k8s.metrics.pkg.apis.external_metrics.v1beta1.ExternalMetricValueList": {
            "$ref": "#/definitions/io.k8s.metrics.pkg.apis.external_metrics.v1beta1.ExternalMetricValue"
          "group": "external.metrics.k8s.io",

Is there anything else I can test to verify that this works?

I'll be working on a PR for the k8s-prometheus-adapter next.

johanneswuerbach commented 4 years ago

Draft PR is ready https://github.com/DirectXMan12/k8s-prometheus-adapter/pull/317

lilic commented 4 years ago

As the linked PR is working correctly @s-urbaniak can you please have a look? Thanks!

s-urbaniak commented 4 years ago

/lgtm

s-urbaniak commented 4 years ago

awesome, thank you! :tada:

lilic commented 4 years ago

@s-urbaniak you need to add the approve label as well: /approve

s-urbaniak commented 4 years ago

/approve

s-urbaniak commented 4 years ago

Thank you @lilic for the bump, I indeed missed the approval :+1:

k8s-ci-robot commented 4 years ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johanneswuerbach, lilic, s-urbaniak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubernetes-sigs/custom-metrics-apiserver/blob/master/OWNERS)~~ [s-urbaniak] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment