Internal Server Error After Upgrading To Kubernetes 1.23

[jimitkr]

I recently upgraded my Kubernetes version to 1.23 and have been getting the following error on the custom metrics api since then:

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
Error from server (InternalError): an error on the server ("Internal Server Error: \"/apis/custom.metrics.k8s.io/v1beta1\": the server could not find the requested resource") has prevented the request from succeeding

Kubernetes api-server has the following error:

E0209 05:15:17.882999      11 controller.go:116] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 500, Body: Internal Server Error: "/openapi/v2": the server could not find the requested resource
, Header: map[Content-Length:[87] Content-Type:[text/plain; charset=utf-8] Date:[Thu, 09 Feb 2023 05:15:17 GMT] X-Content-Type-Options:[nosniff]]

My API service is configured as follows:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: prometheus-adapter
    namespace: monitoring
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

What am i doing wrong?

[k8s-ci-robot]

This issue is currently awaiting triage.

If custom-metrics-apiserver contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[olivierlemasle]

Hi @jimitkr,

It looks you're using prometheus-adapter, right? Which version are you using? How did you install it?

[jimitkr]

@olivierlemasle I'm using Prometheus Adapter v0.6.0 and have installed it via a deployment.yml file in my code. Here's the full deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-adapter
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      name: prometheus-adapter
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        name: prometheus-adapter
    spec:
      containers:
        - args:
            - --cert-dir=/var/run/serving-cert
            - --config=/etc/adapter/config.yaml
            - --logtostderr=true
            - --metrics-relist-interval=15s
            - --prometheus-url=http://prometheus-service.prometheus.svc:9090/
            - --secure-port=6443
          image: <image_uri>
          name: prometheus-adapter
          ports:
            - containerPort: 6443
          volumeMounts:
            - mountPath: /tmp
              name: tmpfs
              readOnly: false
            - mountPath: /var/run/serving-cert
              name: volume-serving-cert
              readOnly: false
            - mountPath: /etc/adapter
              name: config
              readOnly: false
      serviceAccountName: prometheus-adapter
      volumes:
        - emptyDir: {}
          name: tmpfs
        - emptyDir: {}
          name: volume-serving-cert
        - configMap:
            name: prometheus-adapter-config
          name: config

Prometheus adapter logs:

E0209 14:13:25.053677       1 webhook.go:196] Failed to make webhook authorizer request: the server could not find the requested resource
E0209 14:13:25.053754       1 errors.go:77] the server could not find the requested resource
E0209 14:13:25.058400       1 webhook.go:196] Failed to make webhook authorizer request: the server could not find the requested resource
E0209 14:13:25.058478       1 errors.go:77] the server could not find the requested resource
E0209 14:13:25.061852       1 webhook.go:196] Failed to make webhook authorizer request: the server could not find the requested resource
E0209 14:13:25.061916       1 errors.go:77] the server could not find the requested resource

[olivierlemasle]

/transfer-issue prometheus-adapter

[olivierlemasle]

/triage accepted /assign

[jimitkr]

@olivierlemasle Sorry to bother. Did you have any luck replicating the problem?

[jimitkr]

@olivierlemasle My issue was fixed after upgrading prometheus adapter to v0.10.0