canonn-science / CAPIv2-Strapi

Canonn APIv2
https://api.canonn.tech
GNU General Public License v3.0
41 stars 8 forks source link

Create Cron script to pull data from EDSM #2

Closed derrickmehaffy closed 6 years ago

derrickmehaffy commented 6 years ago

We need to create a cron script to either run within strapi or outside as a cronjob.

Our goal is to get the list of systems and bodies and check their updateTime, if it is less than x amount then we query EDSM to pull the data and update it in the database.

There is also the option of doing so before save, but this will require additional testing to see if it works as intended, and how we will handle errors. See the example file here:

https://github.com/canonn-science/CAPIv2-Strapi/blob/582ce042ae4a3931217fa977b4947398864b61f3/api/body/models/Body.js#L7-L10

I believe we could inject the values before we save, but in this case anytime something is changed, it will ask EDSM for an update. We could use this in conjunction with the cron script

derrickmehaffy commented 6 years ago

Have asked @Purrcat#3019 to take a look at this to see if it can be done in python and called using the strapi cron script.

derrickmehaffy commented 6 years ago

@purrcat259 for the arg parsing:

There are 3 different environments:

The files are located in the following 3 directories (database.json not database.json.example) https://github.com/canonn-science/CAPIv2-Strapi/blob/development/config/environments/development/database.json.example (config/environments/development/database.json)

https://github.com/canonn-science/CAPIv2-Strapi/blob/development/config/environments/staging/database.json.example (config/environments/staging/database.json)

https://github.com/canonn-science/CAPIv2-Strapi/blob/development/config/environments/production/database.json.example (config/environments/production/database.json)

The following is what we need to grab:

Obviously in this script the table name systems is static. So for passing arguments to the script file something like:

python3 system_update_edsm.py development python3 system_update_edsm.py staging python3 system_update_edsm.py production

derrickmehaffy commented 6 years ago

After we get the systems script done, then I can pass along the info for the bodies (which will be much much larger and a tad more complex)

derrickmehaffy commented 6 years ago

Script created and runs on each new system added. A new issue will be opened for the cron task