k1m0ch1 / BiBiT

the yogyaonline.co.id, alfacart and klikindomaret scrapper
12 stars 2 forks source link

automated clean up backup database so it won't makes the storage full #22

Open k1m0ch1 opened 5 months ago

k1m0ch1 commented 5 months ago

Main reason to backup the database because the sqlite if not closed while is being used will makes the file corrupt, so backup is like precaution if the database fail to store.

The script to backup is pretty simple, it looks like this

#!/bin/bash

timestamp=$(date +%Y%m%d_%H%M%S)

cd /root/
cp crawl.db /root/workspace/indiemart/backup-indiemart/crawl.db.$timestamp
cp indiemart.db /root/workspace/indiemart/backup-indiemart/indiemart.db.$timestamp

and the backup will be running following this crontab rule

image

now the problem it will makes the server full with storage, there is two option:

  1. using object storage to backup the database (usually 5 USD for 250 GB) cheaper at contabo or DO
  2. make the retention script to make sure the server is not full
k1m0ch1 commented 5 months ago

update: I made it like this:

image

at least it will always delete 5 days ago