SEO: Configure `robots.txt` file.

jimtyhurst commented 3 years ago

Expected Behavior

We want the company website to score as high as possible in the Lighthouse SEO scoring.

Actual Behavior

One of the Gatsby Cloud build reports from Lighthouse dinged our code a few points on SEO, saying:

Search engines are unable to include your pages in search results if they don't have permission to crawl them.
...
Blocking Directive Source
x-robots-tag: none

Analysis

I looked at the meta tags in the build. I do not see that x-robots-tag anywhere. Maybe it is none by default? Maybe we need to explicitly set that tag.
See gatsby-plugin-robots-txt as a way to create robots.txt from metadata in gatsby-config.js

craigfisk commented 3 years ago

If I understand correctly (from the first link below), Lighthouse runs on the results of indexing, and you wouldn't want the Gatsby cloud version to be indexed, so what you are seeing is ok, and Lighthouse will work with the https://marmalade.ai version only. Is that the way you see it, too?

1) https://answers.netlify.com/t/disabling-x-robots-tag-header-on-hosted-site/8857 2) https://www.google.com/search?q=%22x-robots-tag%22+gatsby+lighthouse&oq=%22x-robots-tag%22+gatsby+lighthouse&aqs=chrome..69i57.34455j0j7&client=ms-android-google&sourceid=chrome-mobile&ie=UTF-8#scso=_jdJFYeq7C8G2tAa3y7boBQ14:0

https://www.linkedin.com/in/craigfisk https://about.me/craigfisk (650) 336-8770

On Fri, Sep 17, 2021 at 8:23 PM Jim Tyhurst @.***> wrote:

Expected Behavior

We want the company website to score as high as possible in the Lighthouse SEO scoring. Actual Behavior

One of the Gatsby Cloud build reports https://build-1af2efe3-37c6-4b51-b75f-495fbac4d79f.gtsb.io/reports/lighthouse/index.html from Lighthouse dinged our code a few points on SEO, saying:

Search engines are unable to include your pages in search results if they don't have permission to crawl them. ... Blocking Directive Source x-robots-tag: none

Analysis

I looked at the meta tags in the build. I do not see that x-robots-tag anywhere. Maybe it is none by default? Maybe we need to explicitly set that tag.

See gatsby-plugin-robots-txt https://www.gatsbyjs.com/plugins/gatsby-plugin-robots-txt/ as a way to create robots.txt from metadata in gatsby-config.js http://./gatsby-config.js

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Marmalade-ai/company-website/issues/130, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAZH6WWBH6TMLZOF67ZEYDUCPSXVANCNFSM5EITZ2PQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

jimtyhurst commented 3 years ago

Yes, I was assuming that whatever was in the branch build would be deployed, but it makes sense that it would be different in this case. I will leave this issue open until we deploy the new company website to production, so we can verify that the new site is indexed correctly.

Marmalade-ai / company-website-old