This is preparation to adding support for Creodias (#57).
The base Ingester class is modified so that it can work with crawlers which return the raw attributes of the datasets, not just the download URL. This will be useful when harvesting from APIs which can return lists of metadata about datasets.
I also made a small modification, so that scheduled threads are actually cancelled when the main process is stopped.
This is preparation to adding support for Creodias (#57).
The base Ingester class is modified so that it can work with crawlers which return the raw attributes of the datasets, not just the download URL. This will be useful when harvesting from APIs which can return lists of metadata about datasets.
I also made a small modification, so that scheduled threads are actually cancelled when the main process is stopped.