search

DOI-ONRR / nrrd

ONRR open data site hosted on cloud.gov
Other
13 stars 5 forks source link

Delete sitemap.xml file and update robots.txt file with correct sitemap URL and disable archived site #3379

Closed eelzi-ONRR closed 2 months ago

eelzi-ONRR commented 2 months ago

NRRD already had a sitemap that's autogenerated using Gatsby. It automatically updates when we add new pages so we should keep that one and remove the one we added in Jan 2024. Need to update the robots.txt file with the sitemap link we are keeping.

Get rid of this sitemap: https://revenuedata.doi.gov/sitemap.xml Keep this one (no action needed to keep or update): https://revenuedata.doi.gov/sitemap/sitemap-0.xml

And archive links are still showing up in search results. Need to check that our robots.txt is correctly set up for that and/or try another method

eelzi-ONRR commented 2 months ago

@greensomes robots-dev.txt has been added via PR #3405