Elasticsearch pilot not syncing on schedule

jetstack / navigator

Managed Database-as-a-Service (DBaaS) on Kubernetes

Apache License 2.0

271 stars 31 forks source link

Elasticsearch pilot not syncing on schedule #212

Open munnerz opened 6 years ago

munnerz commented 6 years ago

Looking at the logs below, it appears the elasticsearch pilot sometimes does not sync on its schedule (every 10s).

This could either be due to the sync loop blocking due to lack of timeouts somewhere, or the genericpilot or scheduledWorkQueue having some kind of issue.

Needs investigation.

https://storage.googleapis.com/jetstack-logs/pr-logs/pull/jetstack_navigator/162/navigator-e2e-v1-9/90/artifacts/test-elasticsearchcluster-1516211071-11552/dump.txt

/kind bug

munnerz commented 6 years ago

I think this is due to rate limiting in our work queue - when the Pilot is starting, the sync fails a lot due to the ES client not being available thus causing the back off to balloon.