search

sgrimes / googlesitemapgenerator

Automatically exported from code.google.com/p/googlesitemapgenerator
Apache License 2.0
0 stars 0 forks source link

robots.txt syntax ignored (cakephp with modrewrite) #16

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. put Disallow: /example/ in robots.txt 
2. or put  /example/  in Excluded URL patterns
3. what it works if you put /example/* in Excluded URL patterns

What is the expected output? What do you see instead?

it is expected that the directory /example/ to be ingnored and excluded in
the sitemap

What version of the product are you using? On what operating system?
sitemap_linux-i386-beta1.tar.gz

Please provide any additional information below.
tested on two servers ubuntu 8.10 intrepid also on jaunty
also seems that robots.txt is parsed (i watched the logs)

Original issue reported on code.google.com by map...@gmail.com on 23 Feb 2009 at 10:14

GoogleCodeExporter commented 8 years ago 
Just want to make things clear:

>> 1. put Disallow: /example/ in robots.txt 
If /example/some-page appears before robots.txt is updated. This has no effect.

>> 2. or put  /example/  in Excluded URL patterns
Yes, this is an expected behavior.

>> 3. what it works if you put /example/* in Excluded URL patterns
This works. Right?

Original comment by ma...@google.com on 25 Feb 2009 at 5:27

GoogleCodeExporter commented 8 years ago 
with latest beta i check now if it works 

1) point seems to me that robots.txt rules are now working /example/
and /example/2 are now ignored and not added in latest beta (from 25 Feb)
2) OK but i thought that the same rules from robots.txt are applied 
so it should be more consistent to apply the same rules as in robots.txt
/example/ to be like /example/* 
3. yes i was just notice you that i was expecting /example/ to ignore all 
subdirs 
/example/2 ...

Original comment by map...@gmail.com on 26 Feb 2009 at 9:23

GoogleCodeExporter commented 8 years ago 
Experiencing the same problem: /eDM/ and /WDSearch/ are both disallowed in 
robots.txt.  Configured GSG to exclude /eDM/* and /WDSearch/* and URLs still 
included.  Configured GSG to exclude /eDM/ and /WDSearch/ and URLs still 
included.
Using sitemap_win-beta1-20091231.msi on Windows 2003

Original comment by johnlgoo...@gmail.com on 5 Mar 2010 at 10:39