In mongo db have a new collection to track our data sources with date when the pipeline against it was run so that we can track when we scrapped the data recently .
Source Id
Source Name
Source Link
Source description.
Last run datetime. (Tells when was this data last scrapped)
Raw data location (Azure file storage/ blob storage)
In mongo db have a new collection to track our data sources with date when the pipeline against it was run so that we can track when we scrapped the data recently .