Open arildm opened 9 months ago
Another approach that would probably benefit search bot indexing is #57 "Search query as url parameters", especially including the page number. If I understand correctly, the bot can then navigate through the result list and reach all the work urls by itself.
Google (and other bots) might not easily find all the urls and index them when crawling the site, since most works and terms are only listed after searching or paging/scrolling, which I doubt the bots will be doing.
Background:
robots.txt
/om/
pages. We can combine the two sitemaps with a sitemap index