search

Jordan141 / articleblog

An article blog written using a myriad of web technologies.
2 stars 0 forks source link

Meta tags #233

Open michaelKurowski opened 3 years ago

michaelKurowski commented 3 years ago

We need SEO related metatags

Jordan141 commented 3 years ago

Relevant information after #234

All meta directives (robots or otherwise) are discovered when a URL is crawled. This means that if a robots.txt file disallows the URL from crawling, any meta directive on a page (either in the HTML or the HTTP header) will not be seen and will, effectively, be ignored.

In most cases, using a meta robots tag with parameters "noindex, follow" should be employed as a way to to restrict crawling or indexation instead of using robots.txt file disallows.

It is important to note that malicious crawlers are likely to completely ignore meta directives and as such, this protocol does not make a good security mechanism. If you have private information that you don’t want to make publicly searchable, choose a more secure approach, such as password protection, to keep visitors from viewing confidential pages.

You do not need to use both meta robots and the x-robots-tag on the same page – doing so would be redundant.

Copied from Meta Tags