Open zdodson21 opened 2 months ago
https://www.google.com/search?client=firefox-b-1-d&q=meta+robots+noindex
Go through this and implement anything that seems helpful to block scraping of this website. Also specify with comments in HTML what is there to block scraping, so it can be removed later if needed.
https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
zdodson21
commented
2 months ago
https://www.google.com/search?client=firefox-b-1-d&q=meta+robots+noindex
Go through this and implement anything that seems helpful to block scraping of this website. Also specify with comments in HTML what is there to block scraping, so it can be removed later if needed.