search

lmolteno / nsn

NCEA Standards Navigator
https://www.notion.so/Fast-access-NZQA-d4f21847f9174bc2954bd6a3e8205363
1 stars 1 forks source link

Have a method of setting a flag so that the database gets refreshed #37

Closed lmolteno closed 3 years ago

lmolteno commented 3 years ago

e.g. Have a "flags" table, which has two columns "name", "status". If a "refresh", "True" row is added, then the scraping script will go through and refresh the database on the next run through.

Also, to improve uptime, maybe only clean the database just before we enter the new data in.

lmolteno commented 3 years ago

Currently I do this by deleting the subjects rows, but this is bad

lmolteno commented 3 years ago

Now there's a set_flag.sh script that you can run e.g. ./set_flag refresh, and the next check of the checker script will refresh the database ./set_flag rescrape will tell the script to rescrape entirely.

lmolteno commented 3 years ago

There are problems when wanting to update custom content (#30), where the modified JSON file isn't within the docker image. Fix this by using a docker-compose volume.