search

ansible-community / community-website

Ansible Community website (WIP)
https://ansible-community-website.readthedocs.io
Creative Commons Attribution Share Alike 4.0 International
14 stars 25 forks source link

Add robot meta tag to source #322

Closed wbentley15 closed 9 months ago

wbentley15 commented 11 months ago

Please add the following HTML tag to all website pages:

oraNod commented 11 months ago

@wbentley15 hi. it looks like the tag wasn't included in your comment. could you please update that?

wbentley15 commented 10 months ago

My bad...here it is (please remove the side bars and add proper tags): [ meta name="robots" content="none" ]

oraNod commented 10 months ago

@wbentley15 I'd just like to double check and maybe get the reasoning here. The content="none" directive would prevent the site from being returned in search results. Is this really what we want to do?

https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#directives

I'm speculating but is the motivation so that searches for "Ansible" would point to the Red Hat site rather than the community site?

oraNod commented 10 months ago

This also relates to #355

wbentley15 commented 9 months ago

@oraNod I was asked to include these tags and I am sure they did not want to have the effects your described. The intended point was to keep bots off the site. As long as we have that covered, this issue can be marked as closed/ignored.

oraNod commented 9 months ago

Thanks for the explanation @wbentley15 My understanding of the robots metadata is limited so my speculation in the previous comment might not have even made sense. I'll ask some colleagues who are a lot more versed in this area and get back to you soon.

oraNod commented 9 months ago

I've reached out to the very helpful JP Sherman within Red Hat for advice on that meta and copied you on the thread @wbentley15 - JP confirmed my thoughts that meta name="robots" content="none" would have the effect of keeping all bots off the site, which would have negative impact on search results for the ansible.com domain. JP also gave some guidance on how we can filter bots via robots.txt and keep malicious and unwanted bots off the site, which should be in keeping with the intent of the request.

I've opened #385 to combine this issue with another and come up with a plan for robots.txt before launch. I'll go ahead and close this issue but please feel free to reopen or let me know if I've got anything wrong here. Thanks.