Closed sagargg-zz closed 4 years ago
There was no /robots.txt. When the crawler doesn't find robots.txt the next matched /:organization route throws a 404 error. So, adding robots.txt as a middleware function. For now, it is allowing all paths.
/robots.txt
robots.txt
/:organization
There was no
/robots.txt. When the crawler doesn't findrobots.txtthe next matched/:organizationroute throws a 404 error. So, addingrobots.txtas a middleware function. For now, it is allowing all paths.