search

tubackkhoa / gbif-dataportal

Automatically exported from code.google.com/p/gbif-dataportal
0 stars 1 forks source link

Prepare a strategy for including a META TAG/FILE in the mirror WAR to control Google's crawler access rate #62

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
As mirrors have been reporting problem with Google's web crawler access rate, 
we need to come 
up with a strategy to be able to control Google's crawler access to the 
different mirrors.

Google offers two ways to validate a web site in order for a person to be able 
to access it's 
crawling management module.

2 approaches:

1) Create individual files for each of the mirrors. Then we will validate those 
files in the 
management console.

2) Include a META TAG with the Key that google provides, then we will be able 
to validate against 
that metatag. For this, a property needs to be defined for each mirror on the 
portal.properties 
file. This approach will be similar to the one we use for the Google Maps Key 
each mirror has to 
un-document.

Original issue reported on code.google.com by josecua...@gmail.com on 12 Oct 2009 at 1:52