Enlarge the production DB volume

[mfocko]

We are slowly running out of the space on the production DB volume, therefore we need to enlarge it. Skimmed through the API code, it should be possible to not drop any tasks during the “enlargement procedure”.

• [x] Try it on stage
  • [x] Scale down workers
  • [x] short-running
  • [x] wait for the queue
  • [x] long-running
  • [x] Redeploy database with enlarged volume
  • [x] Rescale postgres
  • [x] Verify the PVC size
  • [x] OpenShift
  • [x] Within the postgres pod
  • [x] Check integrity of the data
  • [x] Scale the workers back up
  • [x] Trigger packit/hello-world
  • [x] Verify it works
• [x] Showdown!
  • [x] Scale down workers
  • [x] short-running
  • [x] wait for the queue
  • [x] long-running
  • [x] Redeploy database with enlarged volume
  • [x] Rescale postgres
  • [x] Verify the PVC size
  • [x] OpenShift
  • [x] Within the postgres pod
  • [x] Check integrity of the data
  • [x] Scale the workers back up
  • [x] Trigger packit/hello-world
  • [x] Verify it works
• [x] Remove banners
  • [x] Matrix
  • [x] Slack
• [x] Remove the status notice
• [x] (optional) Rant if needed… Canceled in the end

[mfocko]

Rescaling packit-service pod, it appears that even read-only API endpoints fail on postgres redeployment /o\

[mfocko]

redeployed, :binoculars: Sentry

[mfocko]

refreshed dashboard few times to make trip, didn't reappear