Problem with editing robot.txt

[rost-royse]

After installing the plugin incorrectly edited robot.txt file. 2.0.1.0 version Episerver CMS 7.5 Its a plugin text area

User-agent: * Disallow: /EPiServer/CMS/ Disallow: /Util/ Disallow: /U123til/

After editing, it shows

Sucessfully saved robots.txt content. All requests to http:///robots.txt will serve this content

(its a project robot.txt )

User-agent: * Disallow: /EPiServer/CMS/ Disallow: /util/ Disallow: /MenuBands1/ Disallow: /MenuBands2/ Disallow: /MenuBands3/ Disallow: /MenuBands4/ Disallow: /offers/ Disallow: /main-menu/ Sitemap: /sitemap_sitepages/

project logs loading and switching

[markeverard]

Ok I've not see any issue like this before, and I'm not quite clear on what the issue is precisely? Is it that the data you are expecting to be saved isn't? (and instead you see some different robots.txt config?)

Those 404's are definitely a problem, though I can't work out why they would be 404's if the user interface itself loads. That shows the routing is correct. Is there anything else you think is relevant?

Robots Txt data is saved in the DDS (and so in the tBlBigTable) in SQL server. perhaps have a look there to see what has been saved.

[rost-royse]

@markeverard The text area of the plugin must contain the data already existing robot.txt file? Data from the text area successfully save in tblBigTable. But when i swiched site the data in text area doesn't change(always displays the content of the first on the site list)

[markeverard]

Ok, so the reason the text doesn’t change when you switch sites is because the AJAX call you listed to the KeyChange action returns a 404.

It is this function and server call that should pass the current site (as selected in the drop down) and return the specific robots.txt

So it’s the 404 that are the issue. Do you have anything else custom in your build around routing? Have you tried the package on an Alloy / default 7.5 site to see if you can replicate the issue.

[ymavi]

Hi Mark, If the Robots.txt content saved in tblBigTable, how to propagate those changes to the physical "Robots.txt" file on the pysical drive. I am using Episerver.CMS version 10.10.0 and POSSIBLE.RobotsTxtHandler version 3.0.0.0. It seems like the when I select the site and save the Robots.txt content it gets saved in the DB, but doesn't reflect in the physical file. Do I need to do any additional steps her. Step 1: Saved the Robots.txt

Step 2: When I access the file using the URL provided, it shows the content of the physical file and not the one I saved.

[markeverard]

Hello @ymavi

I think the issue is the fact that you do have a physical robots.txt file. Delete this and I think you will see the content you expect.

The RobotsTxtHandler module registers a route for /robots.txt in the .NET routetable, and dynamically creates the robots.txt output. I suspect that the fact you have a physical robots.txt file means the .NET / IIS routes and serves the physical file in preference to the content dynamically generated by the module

[ymavi]

@markeverard Thanks Mark. Your suggestion worked and I deleted the physical "Robots.txt" from the site.

[Deepa-Puranik-zz]

We have 3rd party package which is POSSIBLE RobotsTxtHandler for managing our robots.txt file in our website. And there is this error "The page cannot be displayed because an internal server error has occurred." when we update content or click on save button in all the environments of Episerver and not in local development environment where it saves successfully.

This is error log details:

Operation name GET RobotsTxtAdmin/Edit

ErrorMessage Incorrect syntax near the keyword 'Key'.

SELECT Id, ItemType FROM [dbo].[VW_POSSIBLE.RobotsTxtHandler.RobotsTxtData] WHERE Key is null

Event time 8/28/2019, 11:12:05 AM (Local time)

Message Object reference not set to an instance of an object.

Exception type System.NullReferenceException

Failed method POSSIBLE.RobotsTxtHandler.UI.Controllers.RobotsTxtAdminController.Edit

Telemetry type exception

Operation name GET RobotsTxtAdmin/Edit

Problem Id System.NullReferenceException at POSSIBLE.RobotsTxtHandler.UI.Controllers.RobotsTxtAdminController.Edit

System.NullReferenceException:

at POSSIBLE.RobotsTxtHandler.UI.Controllers.RobotsTxtAdminController.Edit (POSSIBLE.RobotsTxtHandler, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null)

at lambda_method (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)

at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod (System.Web.Mvc, Version=5.2.7.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)

at System.Web.Mvc.ControllerActionInvoker+<>c__DisplayClass24_0.b__0 (System.Web.Mvc, Version=5.2.7.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)

at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter (System.Web.Mvc, Version=5.2.7.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)

at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter (System.Web.Mvc, Version=5.2.7.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)

at System.Web.Mvc.ControllerActionInvoker.InvokeAction (System.Web.Mvc, Version=5.2.7.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)

According to this link https://github.com/markeverard/POSSIBLE.RobotsTxtHandler/issues/11 this error will be resolved if we delete the physical file if it’s available in website. And we tried this but this issue is still there even if file is removed if physically present.

Please help.

[markeverard]

Hello Deepa,

From what I understand,

The module saves Robots content successfully in your local environment. However n any other test or production environment it returns a server error with the stacktrace provided on save.

This feels to me like it is data and dynamic data store related (which is why it only appears in certain environments)

Is it possible for you to get a copy of the test or live database locally to see if that will recreate the issue?

I don’t think this is related to #11, but to confirm you don’t need a physical robots.txt file in your solution / website. This modules maps the ~/robots.txt file root to a controller.

[whyleee]

I had the same issue and I found why it happened for me.

There is a bug in Episerver SqlServerDataStoreProvider.GetLoadCommand() method: it doesn't wrap the column name in square brackets when constructing WHERE {0} is null expression and Key is a reserved word in SQL Server - that's why SQL exception occurs.

Same method generates equality expression properly escaped: WHERE [{0}] = @{0}. RobotsTxtHandler always passes site name for Key parameter in normal conditions, so in an ideal world it should never hit that Episerver bug with null expression.

In my case we had IIS rewrite redirects configured to add trailing slash to all URLs except Episerver, but we forgot to exclude "/modules/*" paths from that rule. This resulted in 301 redirect for RobotsTxtHandler form submissions on save, clearing all form values to null and thus hitting that nasty Episerver DDS bug.

So my solution was to exclude "/modules/*" paths from rewriting.

[markeverard]

Great catch @whyleee, and thanks for commenting.

Any ideas of anything the module could do to make this easier to discover? I presume locking down the Edit actions to be HttpPost could help in someway.

[whyleee]

Nice idea to mark Edit actions HttpPost-only 👍 Easy update and the error message helps figuring out the problem faster.