As part of our efforts to improve data curation, we need a clear understanding of our current data sources and the process for integrating new ones. The first step is to create a new page at /sources where we can monitor and manage our data sources more effectively.
Proposed Solution
Create a simple view, such as a table, that shows:
Each data source
The timestamp of when the source was last scraped
The number of documents we have for each source
[your suggestions]
We should be able to get this information by querying Elasticsearch.
kouloumos
commented
3 months ago
As part of our efforts to improve data curation, we need a clear understanding of our current data sources and the process for integrating new ones. The first step is to create a new page at
/sourceswhere we can monitor and manage our data sources more effectively.Proposed Solution
Create a simple view, such as a table, that shows:
We should be able to get this information by querying Elasticsearch.