Add robots.txt

[herzog31]

• Add draft.robots.txt as a blueprint for customers.

Test URLs:

• Before: https://main--aem-boilerplate-commerce--hlxsites.hlx.live/
• After: https://robots2--aem-boilerplate-commerce--hlxsites.hlx.live/

[aem-code-sync[bot]]

Hello, I'm the AEM Code Sync Bot and I will run some actions to deploy your branch and validate page speed. In case there are problems, just click a checkbox below to rerun the respective action.

• [ ] Re-run PSI checks
• [ ] Re-sync branch

Commits

* [7ec4383](https://github.com/hlxsites/aem-boilerplate-commerce/commit/7ec43832b61933e5bdc1bab881e169071cda2152) :arrows_counterclockwise: (latest) * [24c39a4](https://github.com/hlxsites/aem-boilerplate-commerce/commit/24c39a49b56963a15e2cae8749df08175d74ded0) :white_check_mark: * [f372718](https://github.com/hlxsites/aem-boilerplate-commerce/commit/f37271860df6b90237af9225b0afd4545637925f) :white_check_mark: * [e70fe75](https://github.com/hlxsites/aem-boilerplate-commerce/commit/e70fe75b644fbf36c31790d99f9e53df12f8e02b) :white_check_mark: * [17ecd05](https://github.com/hlxsites/aem-boilerplate-commerce/commit/17ecd059ee3f8e1a6afcdd07a4b16a302ec034c2) :white_check_mark: * [32acbcf](https://github.com/hlxsites/aem-boilerplate-commerce/commit/32acbcf379f639f76488b6dfd3911767b74f979a) :white_check_mark:

[aem-code-sync[bot]]

	Page	Scores	Audits	Google
:iphone:	/
:desktop_computer:	/

[herzog31]

@hannessolo @dicagno Can you think of anything else that we should add to this file?

[hannessolo]

Does this take into account luma bridge pages? Should it also disallow pages like /account ?

[herzog31]

@hannessolo Good point. All of those pages should already be excluded via metadata <meta name="robots" content="noindex, nofollow"> e.g. https://main--aem-boilerplate-commerce--hlxsites.hlx.live/customer/account. We could move those over to the robots.txt.

[justinconabree]

Hi @herzog31 here's what we've seen are good paths to exclude in our projects:

/drafts/**
/enrichment/**
/experiments/**
/fragments/**
/tools/**
/nav
/footer

Note this was taken from the exclude from helix-query feeding our sitemap so the pattern might be different.

[herzog31]

Thanks @justinconabree, I added some of your suggestions 👍