Closed j7nw4r closed 2 years ago
KasparWe
commented
2 years ago Have you had a look into troubleshooting section?
Might be this issue https://github.com/kedacore/keda-docs/blob/main/content/troubleshooting/proxy-network.md#solution-for-managed-kubernetes-services
j7nw4r
commented
2 years ago I have. The status of the apiserver seems fine.
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apiregistration.k8s.io/v1","kind":"APIService","metadata":{"annotations":{},"labels":{"app":"environments","app.kubernetes.io/instance":"keda","app.kubernetes.io/name":"v1beta1.external.metrics.k8s.io","app.kubernetes.io/part-of":"keda-operator","app.kubernetes.io/version":"2.7.1"},"name":"v1beta1.external.metrics.k8s.io"},"spec":{"group":"external.metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"keda-metrics-apiserver","namespace":"keda"},"version":"v1beta1","versionPriority":100}}
creationTimestamp: "2022-04-28T20:41:08Z"
labels:
app: environments
app.kubernetes.io/instance: keda
app.kubernetes.io/name: v1beta1.external.metrics.k8s.io
app.kubernetes.io/part-of: keda-operator
app.kubernetes.io/version: 2.7.1
name: v1beta1.external.metrics.k8s.io
resourceVersion: "112946475"
uid: 314c8c9b-5315-4ebd-9325-e1c362a00738
spec:
group: external.metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: keda-metrics-apiserver
namespace: keda
port: 443
version: v1beta1
versionPriority: 100
status:
conditions:
- lastTransitionTime: "2022-05-18T16:57:40Z"
message: all checks passed
reason: Passed
status: "True"
type: AvailableThe suggestion to allow 6433 doesn't really make sense to me. We're not operating a private cluster, and I would assume that the metrics server would be internal to the cluster?
j7nw4r
commented
2 years ago I turned up the logging on keda-metrics-apiserver to 1.
I've noticed that periodically, when querying kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1 the server responds with Error from server (ServiceUnavailable): the server is currently unable to handle the request.
This makes sense. The HPA's sometimes shows a reading.
j7nw4r
commented
2 years ago The keda-metrics-apiserver seems to be intermittently unavailable? Example shown here: https://asciinema.org/a/yC6MkXiL66SAG5IE4GMoG72R4
There are no errors in the logs for the keda-metrics-apiserver for these dropped requests.
JorTurFer
commented
2 years ago hey @johnathan-walker Is the KEDA pod running out of resources? How many ScaledObject do you have?
j7nw4r
commented
2 years ago Total, we have 8. Each ScaledObject has two triggers (one for rabbit, one for cron). I've been able to replicate this with just one SO (two triggers).
As for resources for the KEDA pod, I'm using the resource limits defined in the yamls here.
j7nw4r
commented
2 years ago resources:
limits:
cpu: 1000m
memory: 1000Mi
requests:
cpu: 100m
memory: 100MiNo no, we know about problems with one thousand SO, 8 is not a problem at all.
If you enable log level 1 in metrics server and do again the manual requests, do you see any error in the metrics-server pod?
I mean, every request should be traced in the metrics-server log if I'm not wrong
just in case, to change log level in the metrics server, you should modify the arg - '--v=0' with - '--v=1'
j7nw4r
commented
2 years ago Eh, no errors with log --v=1. My next step is to try to validate that all of the traffic is actually getting to the apiserver.
JorTurFer
commented
2 years ago Sorry for the slow response.... How many (KEDA) instances do you have? I have been checking the video you sent again and that behaviour suggests me 2 different scenarios.
kubectl get pods -l app=keda-metrics-apiserver -n kedaI'm assuming you installed KEDA with default values, if not, just update the namespace and the labelSelector. This is an example of the response:
NAME READY STATUS RESTARTS AGE
keda-metrics-apiserver-c94fdcc8f-tqzpq 1/1 Running 0 7d6h@JorTurFer Thanks. That actually was the issue.
We use kustomize for our manifest deployments. We added a common label that was overriding the label of the service. This caused the operator to be included in the service selection (and to fail).
I tried this on both minikube and a new gke cluster and couldn't for the life of me figure it out. Thanks a lot.
Maybe we can add a warning about the label so that it's not accidentally overridden?
JorTurFer
commented
2 years ago Do you mean in the installation docs? Yes, for sure, every improvement in the docs is more than welcomed 😄 Are you willing to contribute with that?
j7nw4r
commented
2 years ago Yes in the installation. I'll provide a pr.
stale[bot]
commented
2 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.
stale[bot]
commented
2 years ago This issue has been automatically closed due to inactivity.
Report
When creating a ScaledObject using any scaler (tested on both cron and rabbitmq) scaling falls back to the
fallbackconfiguration and the following error is seen in the HPA:ScaledObject yaml
Expected Behavior
ScaledObject creates an HPA that properly reads scaling metrics.
Actual Behavior
ScaledObject seems to fall back to the
fallbackconfiguration.HPA reports:
Steps to Reproduce the Problem
Logs from KEDA operator
KEDA Version
2.7.1
Kubernetes Version
1.23
Platform
Google Cloud
Scaler Details
Cron and RabbbitMQ
Anything else?
Keda Metrics API Server
Kubernetes Metric Server