Allow Data studios to read daily stats from Zen

[conoramurphy-zz]

Dojo data regularity

Since we are reading stats and data from the data warehouse, making it as current as possible helps the dependability for the community team a lot.

At the least, we should guarantee and add last day of the month backup. So it's always fresh data when they report on the first of each month.

If it was feasible to do daily backups that would be best as it's simpler for everyone to keep track of.

Suggestion

It was suggested in a meeting on stats pipelines that we could move to dailys using a system lightly detailed here https://docs.google.com/document/d/1T8Qat1kIpJS3nCVtZnqxAWPm6kzQvj9tQVyg-S5-_PM/edit

Monitor

• [ ] Runs properly daily
• [ ] Save the weekly specific every sunday
• [ ] Rotate the /daily after 90 days
• [ ] Rotate the /weekly after 365 days

[Wardormeur]

After further investigation, solution is :

• [x] Remove --jobs from pg_dump, paralyzing the db
• [x] Move recent dumps to /daily
• [x] Create another dump per week into /weekly
• [x] Change recovery of dumps by warehouse to /daily instead of /
• [x] Add a s3 lifecycle rule to clean-up /daily for anything older than 90 days (270 dumps on daily)
• [x] Add a s3 lifecycle rule to clean-up /weekly for anything older than 53 weeks (a year) (56 dumps)
• [x] Change frequency of jobs to daily
• [x] Move current dumps to /archive (putting them into weekly would be awkward)

[Wardormeur]

Done and added monitoring for the next year to look at