danrademacher
commented
9 months ago Wow, yeah, that's a lot of big files we have lying around! So we're saving a full Planet every day?
A few questions:
- Is the practice at OSM also to delete all backups older than a month?
- Would there be any reason to take a more nuanced approach like some backup software like this:
- Daily Planets for the last month
- Monthly Planets for the past year (say from the first day of each month)
- Annual Planets for the full history of the project (say from the first day of each year)
That approach would be more challenging to implement, but it would be a bit more balanced between economy, backup safety, and keeping a history of the project.
Rub21
1ec5
batpad
After examining the S3 costs, I discovered that the storage expense is 10 bucks per day, with the highest cost attributed to storage. Consequently, it appears that we are retaining some files that are no longer necessary. For example:
Imposm Expired Files: These files are no longer needed. Once Imposm pushes data to S3, the tiler cache retrieves those files. To clean the cache, it only removes the files that were uploaded a minute ago.
Planet Files: It is necessary to clean and remove files older than a month. Access the files here.
Planet Full History Files: Similar to Planet files, we need to clean and remove files older than a month. Access the files here.
DB (Web) Backups: We took care of cleaning up these files a month ago and implemented a script that removes files older than 1 month.
@batpad @danrademacher Confirm the decision to remove at least the planet replica and full planet files older than a month, as well as expired files because they are no longer required, lLater, we can replicate, as we did for DB backups, to remove older files from S3.