Multi file sitemap (but a better one)

[MeowMeowVenom]

The current sitemap, has all the links included (like users, tags, and posts). It would be good if the following happens

sitemap.xml (this should link only the sitemap of users, tags, posts)

sitemap-user.xml (Sitemap just with the users) sitemap-tags.xml (Sitemap just with tags) sitemap-post.xml (Sitemap with just posts)

This is a good addition, because I dont want to submit my user dierctory to google search console. Its like unwanted 4000 links.

[luceos]

The sitemap.xml is generates based on all best practices from roughly a year or two ago. Unless something changed in those two years I don't see why simplifying a file meant for machines ought to be more easily read by a human, can you clarify?

It's not that you need to submit all the entries in sitemap.xml to search engines as just that file is sufficient for them to automatically get all links, even compressed at that.

If I'm not mistaken you should be able to disable the user links from a setting.

[MeowMeowVenom]

Its just that having users link in sitemap, which the google search console uses, seemed unwanted. That’s it, nothing much. Domestika.org, does something like this (https://www.domestika.org/sitemap.xml)

From: Daniël @.> Sent: Tuesday, September 28, 2021 10:35 PM To: @.> Cc: @.>; @.> Subject: Re: [FriendsOfFlarum/sitemap] Multi file sitemap (but a better one) (#33)

The sitemap.xml is generates based on all best practices from roughly a year or two ago. Unless something changed in those two years I don't see why simplifying a file meant for machines ought to be more easily read by a human, can you clarify?

It's not that you need to submit all the entries in sitemap.xml to search engines as just that file is sufficient for them to automatically get all links, even compressed at that.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/FriendsOfFlarum/sitemap/issues/33#issuecomment-929413827, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO7MZDJWIOD6WAJLJOVBHUTUEHYUJANCNFSM5E4OI5RA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[clarkwinkelmann]

There are two things here:

1) Whether to split the sitemap into multiple files: we already offer options to split the files based on the 50k links limit. This is the industry standard.

2) How to control which links are added to the sitemap. At the moment there is no dedicated setting for user profiles. The implementation will always add links to profiles if guests are allowed to search for users. If guests aren't allowed to search for users, the links aren't added.

I don't think we need to do anything for (1) unless there are new best practices or that there's a clear benefit of splitting files that way. For (2) we should probably add a switch setting at some point because this has been regularly requested. Though most of the people who wanted to remove users from indexing also wanted to disable guest user search, so the current implementation has never been a deal breaker to anyone as far as I know.

[clarkwinkelmann]

Version 2.0 of Sitemap now generates at least one file per resource. The filenames don't contain the resource type, they are just numbered starting with index 0, and then switches to the next index after reaching 50k links or when changing resource.

The user directory concern has been addressed through a new setting controlling user index (or the RemoveResource already available in previous releases).

I'm going to close this issue since the points have been addressed and I don't think we will change this again anytime soon.