Have new collection to track Data Sources

ShelterApp / AddResources

http://shelterapp.org/

11 stars 10 forks source link

Have new collection to track Data Sources #7

Closed prabhushrikant closed 3 years ago

prabhushrikant commented 4 years ago

In mongo db have a new collection to track our data sources with date when the pipeline against it was run so that we can track when we scrapped the data recently .

Source Id
Source Name
Source Link
Source description.
Last run datetime. (Tells when was this data last scrapped)
Raw data location (Azure file storage/ blob storage)

prabhushrikant commented 3 years ago

We now have data-sources collection in mongoDB which tracks which collection was scrapped when in scraper.