The news feeds and sitemaps can be useful by itself - the feeds more than the sitemaps because they include news titles and short snippets. It might make sense to put them also into the WARC files.
But first, it's important to understand what the storage foot print would be as feeds/sitemaps are refetched multiple times per day.
The news feeds and sitemaps can be useful by itself - the feeds more than the sitemaps because they include news titles and short snippets. It might make sense to put them also into the WARC files.
But first, it's important to understand what the storage foot print would be as feeds/sitemaps are refetched multiple times per day.