search

aemsites / 888de

Apache License 2.0
0 stars 0 forks source link

robots.txt files #246

Open helms-charity opened 1 month ago

helms-charity commented 1 month ago

Do we need to add a robots.txt and a stage-robots.txt?

Liron Wulffhart said:

I see in my nightly tests that the prod robots should return 404 but the stage robots should return 200 ... this is how it was on Tridion ... strange that we didn't have a prod robots for this site ...

The staging for robots.txt would need a CDN redirect to stage-robots.txt. We need to know how to configure the files.

Example of a production configuration:

User-agent: *
Allow: /

Sitemap: https://sitemap.888.com/umbrella/888-DE.xml