adamreichold
commented
2 years ago Since we are currently always produce a full consistent snapshot of the datasets, this implies that all sources which are not filtered for in the current run need to be copied/hard-linked into the datasets.new directory.
Add a group setting to sources on which the harvester will filter them, e.g. if given a group via command line or environment variable, so that different periodically triggered harvester systemd services can be defined so that e.g. some sources are harvested daily while others are harvested only weekly without having to implement a general purpose scheduler (and the accompanying need for persistent state) in the harvester.