Reduce data size in the database

CivicTechTO / ttc_subway_times

A scraper to grab and publish TTC subway arrival times.

GNU General Public License v3.0

40 stars 30 forks source link

Reduce data size in the database #32

Closed radumas closed 5 years ago

radumas commented 6 years ago

Having burned through my 12 months of AWS Free tier, running this operation is currently ~$30/month because of the size of the RDS. We need a way to store backups of the database data in compressed monthly files that others can access. Preferably these would be hosted in S3 buckets on AWS. Until then prepare a command to archive a month of all three tables to csv and compress them (and store them in SpiderOak)

radumas commented 6 years ago

Final piece remaining is to code in deleting month of data from database after compressing it. I'm also going to run this on my Raspberry Pi at home, because that's free.