search

knative / serving

Kubernetes-based, scale-to-zero, request-driven compute
https://knative.dev/docs/serving/
Apache License 2.0
5.54k stars 1.15k forks source link

Error in Autoscaler Log after fresh install #10917

Closed Miles-Ahead-Digital closed 3 years ago

Miles-Ahead-Digital commented 3 years ago

Hi,

after a fresh install I get the following error in the autoscaler log:

E0308 18:59:50.604291 1 leaderelection.go:361] Failed to update lock: Operation cannot be fulfilled on leases.coordination.k8s.io "autoscaler-bucket-00-of-01": the object has been modified; please apply your changes to the latest version and try again

What is the reason for this? Stefan

markusthoemmes commented 3 years ago

Probably an eventual-consistency issue that resolves itself. Usually, these are harmless. Are you getting them continuously?

Miles-Ahead-Digital commented 3 years ago

yes approx. every 40 seconds. But I cant tie an error to it directly..

vagababov commented 3 years ago

cc @yanweiguo

What kind of install are you running?

Miles-Ahead-Digital commented 3 years ago

I run the install from https://knative.dev/docs/install/any-kubernetes-cluster/

I'm using a GKE Cluster with Workload Identity enabled and I'm wondering if this is the reason. Can you provide documentation how to setup knative on a cluster using Workload Identity?

These are the parameters I used to create the cluster:

gcloud beta container clusters create "gke-$base" \ --metadata disable-legacy-endpoints=true \ --enable-autoscaling --service-account="$node_sa_email" \ --workload-metadata-from-node=GKE_METADATA_SERVER \ --shielded-integrity-monitoring --shielded-secure-boot \ --addons=HorizontalPodAutoscaling,NetworkPolicy,NodeLocalDNS \ --database-encryption-key="$key_id" --no-enable-basic-auth \ --enable-ip-alias --no-enable-legacy-authorization \ --enable-network-policy --enable-shielded-nodes \ --enable-stackdriver-kubernetes \ --identity-namespace="$base-cluster.svc.id.goog" \ --image-type=COS_CONTAINERD --no-issue-client-certificate \ --machine-type=e2-standard-2 --max-nodes=3 --min-nodes=1 \ --network="net-$base" --subnetwork="subnet-$base" \ --master-authorized-networks="$(curl -s https://icanhazip.com/)/32" \ --enable-private-nodes --master-ipv4-cidr="$mcidr" \

yanweiguo commented 3 years ago

I don't think workload Identity is the reason to cause the leaderelection logs. Basically workload identify controls the permissions of your GKE components to call other google cloud products, like, GCR, GCE API. The leaderelection is not relative.

IMO those logs are standard logs on conflicts which don't indicate an issue. Could you confirm that the scaling functionality is working on your cluster?

Miles-Ahead-Digital commented 3 years ago

the scaling functionality does not work, I found the https://github.com/google/knative-gcp project and will use their GCP specific setup of knative.

Therefore I will close this message.

dkozlov commented 3 years ago

I have the same issue but on AWS