search

10up / distributor

Share content between your websites.
https://distributorplugin.com
GNU General Public License v2.0
629 stars 154 forks source link

Excluding distributed posts from sitemaps #362

Open joshuaabenazer opened 5 years ago

joshuaabenazer commented 5 years ago

Sitemaps provide priority consideration for crawling, but indexing is determined based on the canonicals and indexing tags, not by the sitemap files themselves.

Plugins that generate sitemaps like Jetpack and Yoast do not exclude the distributed posts by default from the sitemaps. The crawlers are pretty sophisticated and if the canonicals are in place, there's no reason to worry, but if there's a significant number of distributed posts, looking from a crawl capacity standpoint, we wouldn't want to waste Google's crawl time on the distributed content and have it prioritized for more important content.

That being said, I feel it would be a good addition to the distributor plugin to handle removal of the distributed posts from the sitemaps out of the box, that are generated by popular plugins like Jetpack and Yoast.

jeffpaul commented 5 years ago

@joshuaabenazer thanks for the feedback, I'm milestoning this into our 2.0.0 release but if you've got time to create a PR that would be warmly welcomed!