NoIndex (SEO)

[jamie9090]

Hi there!

Love this plugin, it gave me single note publishing which is really what I was looking for.

I just need a hand marking my digital garden as noindex so search engines don't crawl it.

Could someone kindly assist with the code for vercel.json to do this? I have a default install, so nothing fancy there.

Cheers, Jamie

[ludwikc]

I've asked the same question on OMG (Obsidian Members Group) Discord, but haven't got the answer yet.

[dayne]

The way to do this is to have a /robots.txt on the website with the following content:

User-agent: *
Disallow: /

Just need to make sure the file is accessible via http://yourdomain.com/robots.txt.

While most major search engines respect the robots.txt directive, some minor or malicious bots may ignore it. Over time, if your site was previously indexed and you add the block later, it may take some time for search engines to remove the indexed pages. If you want to expedite the removal of specific pages or the entire site, you can use tools provided by search engines, like Google's Search Console.

How to do this for your site?

The static site generator, eleventy.js, being used here isn't one I'm very familiar with but I did figure it out. I tried added a robots-test.txt into my garden repo under src/site/robots-test.txt to see if that caused it to show up under /robots-test.txt on the Vercel hosted website - no luck.

Looking deeper I found this blog post on adding robots.txt to eleventy site.

What is also needed is something like below tell the static site generator, eleventy, to pass that file through.

// Put robots.txt in root
eleventyConfig.addPassthroughCopy({ 'src/site/robots.txt': '/robots.txt' });

So I went to around line 540 in my site's .eleventy.js file, right before the userEleventySetup(eleventyConfig); line and added the following above it:

   // Put robots-text.txt in root
   eleventyConfig.addPassthroughCopy({ 'src/site/robots-test.txt': '/robots-test.txt' });

That did the trick and my new robots-test.txt showed up as expected.

Hope that helps.

Warning: I'm not sure if adding that line will be kept gracefully during a template update -- but in theory it should -- but not tested that yet.

Documented this over in my digital garden so if this does help we could probably close this issue (or add them to the docs and then close).

https://hive.mod0.org/tech/obsidian/robots-txt-for-digital-garden/