kostmo / circleci-failure-tracker

A log analyzer for CircleCI. Note that this project is now hosted at pytorch/dr-ci
https://github.com/pytorch/dr-ci
5 stars 2 forks source link

accumulation of open database sessions for materialized view refresh #56

Closed kostmo closed 5 years ago

kostmo commented 5 years ago

At around 6pm on 09/12/2019, the RDS storage was automatically autoscaled upward to 30GB.

Around that time (perhaps just before), database sessions began stacking linearly. A REFRESH MATERIALIZED VIEW command is scheduled to run every 10 minutes. However, these piled up over a period of about 5 hours, until I manually rebooted the database.

Screen Shot 2019-09-13 at 10 13 04 AM

As an attempt to avoid this in the future, I added:

SET SESSION lock_timeout = 3000;

just before the REFRESH MATERIALIZED VIEW CONCURRENTLY [...] command.