Reconciles take a long time

mook-as commented 4 years ago

Describe the bug Apologies for the really vague bug report.

We've been noticing that sometimes resources will take a long time to upgrade. Unfortunately we are not sure yet under what circumstances this occurs.

I've seen this with QuarksStatefulSet not being regenerated, as well as QuarkJobs not creating the Job when triggered.

To Reproduce (As an example):

Deploy a QuarksJob
Trigger it by setting the strategy to now.
Notice that the Job isn't created for a long time

Expected behaviour

The Job (or whatever) is updated soon-ish.

Actual behaviour

The Job (or whatever) is only triggered after a long time (possibly on the order of an hour).

Environment

CF-Operator Version: v6.1.10-0.g7b1a5f2f
KubeCF 2.5.4.
minikube

Additional context

Deleting the controller (e.g. the cf-operator / quarks-job / quarks-secret pod) and letting it restart seems to (at least some of the time) make things happen faster.
Will attempt to fetch logs the next time this happens.

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175135832

The labels on this github issue will be updated when the story is started.

manno commented 4 years ago

There was a bug with updating the status.

cloudfoundry-incubator / quarks-operator

Reconciles take a long time #1171