This adds a mechanism to ensure that we only download and load a dataset if the server says it has been modified since we last downloaded it.
With this in place, we can trigger even the monthly and yearly jobs on a daily basis, ensuring that they are updated as soon as a new version of the dataset is made available (it appears many of these datasets are not updated on a fixed schedule, so this would be useful), without incurring many unnecessary hours of compute time.
This adds a mechanism to ensure that we only download and load a dataset if the server says it has been modified since we last downloaded it.
With this in place, we can trigger even the monthly and yearly jobs on a daily basis, ensuring that they are updated as soon as a new version of the dataset is made available (it appears many of these datasets are not updated on a fixed schedule, so this would be useful), without incurring many unnecessary hours of compute time.
To do