search

SFPC / sfpc.github.io

School for Poetic Computation website.
https://sfpc.io
23 stars 38 forks source link

Create robots.txt #245

Open ibrand opened 1 year ago

ibrand commented 1 year ago

De-index entire archval site from google. robots.txt is the softest way to block the indexing and still might not work in some cases -- to fully get rid of the indexing you'd want to use: <meta name="robots" content="noindex"> -- Explained here: https://stackoverflow.com/a/47083230

But I think this would likely work well enough. It has for my personal site so far. It'll take a little troubleshooting to verify that this worked but I can submit another PR if adjustments need to be made.

One caution: if this does work, google wont recrawl in the future (though i think you can submit a re-crawl request) so best to make sure you actually want the full site de-indexed from the search results.

netlify[bot] commented 1 year ago

Deploy Preview for sfpc failed.

Name Link
Latest commit ebb37613378979a24ffdadeaec6404f629a3a9c7
Latest deploy log https://app.netlify.com/sites/sfpc/deploys/64fb63ba8aeeda00081e4df2