This is the source code for the Canvas Data Loader. The Canvas Data Loader is an example application that downloads your data, and imports it into a Database. The process is completely automated, and is able to handle things like Historical Refreshes, Schema Changes, and the 24-36 hour variance all without issue.
It should be noted although there are better options out there, there isn't a reason why you couldn't use the loader to handle all of your imports everyday. The Canvas Data Loader could for example handle your imports at first, and then later off be handed to a more stable process.
Although this is under the Instructure Repo this is purely an example application, and as such is not fully supported by Instructure.
However, the Canvas Data Support team is happy to field requests about usage in the standard canvasdatahelp@instructure.com email.
The following instructions are for a linux server, but steps 1-5 should work universally. You'll just need to use your systems way of scheduling a repeating task instead of crons if you are not using linux.
cp ./config/default.toml ./config/local.tomlmy_text_editor ./config/local.tomlcargo build --release.crontab -e0 * * * * cd <my_cdl_location> && RUST_LOG=info ./target/release/cdl-runner > /var/log/cdl-log 2>&1Configuration can also be done using environment variables instead of, or in addition to the ./config/local.toml file. For example, you may wish to use environment variables for the API key/secret and use the file for the remaining configuration.
Example:
export cdl__canvasdataauth__api_key=abcdefg123456
export cdl__canvasdataauth__api_secret=123456abcdefg
Possible environment variables:
cdl__canvasdataauth__api_keycdl__canvasdataauth__api_secretcdl__database__db_typecdl__database__url cdl__only_load_finalcdl__rocksdb_locationcdl__save_locationcdl__skip_historical_importsThe Canvas Data Loader is Licensed under MIT.