web robots can access downloads and example links

[GoogleCodeExporter]

Looking at the server access statistics (/log, etc), specifically for FILE 
category, and the metadata (logentity table), I concluded that robots do 
download all the data files and run example queries from time to time (e.g., 
such as the one from Mountain View, US, perhaps Google). Also, our robots.txt 
file is too trivial, and more importantly, does not work at all when cpath2 is 
deployed not as a web domain's root web app... Also, ";jsessionid=..." is 
sometimes part of the log record.

To tune, fix this, I am going to:
- remove robots.txt and the corresponding web controller;
- add either <meta name="robots" value="nofollow,noindex"> (for the /admin 
pages) or <a rel="nofollow" ..> (for file and example links).
- remove ";jsession..." ending from a file name before saving the download 
event to the log db.

Original issue reported on code.google.com by rod...@gmail.com on 5 Nov 2014 at 4:21

[GoogleCodeExporter]

Fixed in the cPath2 sources; - will be deployed on the test server at 
http://pathwaycommons.baderlab.org first

Original comment by rod...@gmail.com on 5 Nov 2014 at 4:28

• Changed state: Started

[GoogleCodeExporter]

Original comment by rod...@gmail.com on 7 Nov 2014 at 10:23

• Changed state: Fixed