Set up a Cron job to archive old games

[adrian-golian]

Set up a process to periodically collect old, inactive games and put them on S3 - clearing space on the instance.

[maciej-pomykala]

From S3 FAQs:

Upon sign-up, new AWS customers receive 5 GB of Amazon S3 Standard storage, 20,000 Get Requests, 2,000 Put Requests, 15GB of data transfer in, and 15GB of data transfer out each month for one year.

So we need to move files in batches (as opposed to one file per request). This is the current solution:

In Cron jobs, accessible at crontab -e, we have:

0 * * * * sh ~/move_game_data.sh >> cron.log 2>&1

In ~/move_game_data.sh, we have:

now=$(date +"%Y.%m.%d_%H")
mkdir game_data/v2/${now}
find game_data/v2/ -maxdepth 1 -mmin +1440 -exec mv '{}' game_data/v2/${now} \;
aws s3 mv game_data/v2/${now} s3://blef-game-data/v2/ --recursive
rm -r game_data/v2/${now}

It works.

I will make it a part of the API deployment. That will probably mean that I will put the above script in the /api/ directory and, using run_api.sh, we will copy the script to the desired destination (does not have to be in ~) and check that our S3 bucket has the appropriate directory for the engine version.

@adrian-golian Any thoughts?

[adrian-golian]

I think this is good. I would only suggest changing the chron to run once a day at midnight UTC and to tar/zip the data before putting it on S3, naming it with the time stamp ($now).

[adrian-golian]

As we talked @maciej-pomykala , the script should be part of deployment, it should though make sure to kill a previous chron job as well as be tested after deployment, in the test yml.

[maciej-pomykala]

As agreed, we will just mention the script in a comment to the issue.

Under crontab -e we will find the following:

0 * * * * sh ~/move_game_data.sh >> cron.log 2>&1

And in move_game_data.sh we will find the following:

now=$(date +"%Y.%m.%d_%H")
mkdir game_data/v2/${now}
find game_data/v2/ -maxdepth 1 -mmin +1440 -exec mv '{}' game_data/v2/${now} \;
zip -r game_data/v2/${now}.zip game_data/v2/${now}
aws s3 mv game_data/v2/${now}.zip s3://blef-game-data/v2/
rm -r game_data/v2/${now}

It works and, in contrast to the earlier solution, only moves to S3 one file per hour.

Requesting closure of the issue.

[adrian-golian]

(Forgot the issue wasn’t closed by automation - no PR) Closing.