Currently, we have the following behaviors given an update interval:
:daily: The file is updated if the current day number is grater than the file day number.
:weekly: The file is updated if a Monday has passed after the file date.
:monthly: The file is updated if the first day of a new month has passed after the file date.
:yearly: The file is updated if the first day of the year has passed after the file date.
The problem are with files that are updated in those intervals, but at the end of the window. For example, if a file is updated every December 30, then RemoteFiles will download it repeatedly until December 30 of the next year.
This commit treats the intervals relative to the current date of the file. Hence, if the file mtime is December 30, 2021, and the update is set to :yearly, then RemoteFiles will only download a new copy after December 30, 2022. Thus, the new behaviors are:
:daily: The file is updated if 1 day has passed from the file date.
:weekly: The file is updated if 7 days has passed from the file date.
:monthly: The file is updated if 30 days has passed from the file date.
:yearly: The file is updated if 365 days has passed from the file date.
A problem with this approach exists if the download tool stamps the mtime with the time in which the file was downloaded instead of the time in which the file was modified at the source. However, both curl and wget seems to keep the original time stamp.
I also modified in this commit how the current time is obtained. It uses now(UTC) instead of now() because the file timestamp (obtained with unix2datetime) will always be in UTC.
ronisbr
commented
2 years ago
Currently, we have the following behaviors given an update interval:
:daily: The file is updated if the current day number is grater than the file day number.:weekly: The file is updated if a Monday has passed after the file date.:monthly: The file is updated if the first day of a new month has passed after the file date.:yearly: The file is updated if the first day of the year has passed after the file date.The problem are with files that are updated in those intervals, but at the end of the window. For example, if a file is updated every December 30, then RemoteFiles will download it repeatedly until December 30 of the next year.
This commit treats the intervals relative to the current date of the file. Hence, if the file
mtimeis December 30, 2021, and the update is set to:yearly, then RemoteFiles will only download a new copy after December 30, 2022. Thus, the new behaviors are::daily: The file is updated if 1 day has passed from the file date.:weekly: The file is updated if 7 days has passed from the file date.:monthly: The file is updated if 30 days has passed from the file date.:yearly: The file is updated if 365 days has passed from the file date.A problem with this approach exists if the download tool stamps the
mtimewith the time in which the file was downloaded instead of the time in which the file was modified at the source. However, bothcurlandwgetseems to keep the original time stamp.I also modified in this commit how the current time is obtained. It uses
now(UTC)instead ofnow()because the file timestamp (obtained withunix2datetime) will always be in UTC.Closes #25