https://thegazelle.org/robots.txt returns HTML Status 500 on the production site. It is supposed to return HTML status 200 along with a valid robots.txt. Issue was first noticed when running Google Lighthouse on the site, and it complained that the site does not have a robots.txt.
Expected Behavior
Current Behavior
Currently, the code has mechanisms in place for displaying a robots.txt for the admin panel (disallow all ), and displaying a robots.txt for the staging site ( disallow all ) as we want neither of those to be indexed on Google. However there is no mechanism in place for the production site as the robots.txt is added as a route in a conditional (ifStaging() is true), thus there is no robots.txt for the main site.
Possible Solution
Add an additional conditional check to the server routes, and add a GET route for robots.txt ifProduction() is true. Robots.txt should allow crawling so that our site can properly be indexed by any search engines that obey robots.txt
https://thegazelle.org/robots.txt returns HTML Status 500 on the production site. It is supposed to return HTML status 200 along with a valid robots.txt. Issue was first noticed when running Google Lighthouse on the site, and it complained that the site does not have a robots.txt.
Expected Behavior
Current Behavior
Currently, the code has mechanisms in place for displaying a robots.txt for the admin panel (disallow all ), and displaying a robots.txt for the staging site ( disallow all ) as we want neither of those to be indexed on Google. However there is no mechanism in place for the production site as the robots.txt is added as a route in a conditional (ifStaging() is true), thus there is no robots.txt for the main site.
Possible Solution
Add an additional conditional check to the server routes, and add a GET route for robots.txt ifProduction() is true. Robots.txt should allow crawling so that our site can properly be indexed by any search engines that obey robots.txt
Steps to Reproduce (for bugs)