Note that the REPEL pipeline does not provide an example of any logic around downloading only new data, because the downloads were quick enough and the update period infrequent enough (every 6 months) that it was worth just doing a full download. For OpenRVF, we have bigger data and more frequent (monthly) updating needs.
Some of these steps can use/adapt existing functions from the training pipeline.
See REPEL2 for the logic we used to create a data processing pipeline for prediction purposes, separate from the model training pipeline: https://github.com/ecohealthalliance/repel2/blob/dev/tar_plans/_targets_prediction_livestock.R
Note that the REPEL pipeline does not provide an example of any logic around downloading only new data, because the downloads were quick enough and the update period infrequent enough (every 6 months) that it was worth just doing a full download. For OpenRVF, we have bigger data and more frequent (monthly) updating needs.