Closed m-appel closed 9 months ago
While I'd argue that it's not a magic number, since the assignment to the days=
parameter should make its meaning pretty clear, we can put it in the configuration, maybe as a generic option (i.e., independent of the PeeringDB crawlers, since we might use CachedSession
in the future for other crawlers.
How about config['cache']['duration_in_days']
? We could also add the directory there:
{
"cache": {
"directory": "tmp/",
"duration_in_days": 6
}
}
yes, that sounds good to me. So if we increase the frequency of dumps, we can just modify the configuration file and we don't have to touch the code
We now read the directory and duration from the config. (I think at some point we should also do something about how we handle the config file :D) I've set the default behavior (i.e., when run without config) to "Do not cache".
Description
The cache of the PeeringDB crawler now expires after six days. Therefore, the weekly dump will always fetch fresh data.
Motivation and Context
Fixes #114 .
Types of changes
Checklist: