Update readiness/liveness probe timings - Githubissues

mozilla / experimenter

A web application for managing user experiments for Mozilla Firefox.

https://experimenter.services.mozilla.com

Mozilla Public License 2.0

115 stars 180 forks source link

Update readiness/liveness probe timings #10790

Open jaredlockhart opened 4 months ago

jaredlockhart commented 4 months ago

We've been seeing a lot of readiness/liveness probe failures in our k8s setup, but every other indication is that our app is functioning healthily so let's update these timings to something more liberal since we know it can take some number of seconds for the canary to start and run migrations, import features, etc

https://github.com/mozilla-services/cloudops-infra/blob/master/projects/experimenter/k8s/charts/experimenter/templates/deployment.yaml#L64-L85

┆Issue is synchronized with this Jira Task

jaredlockhart commented 4 months ago

Let's consider moving migraitons and feature config loading into a separate kubernetes 'job' so that the request handler pods are responsible only for collectstatic and starting gunicorn.