search

tubackkhoa / gbif-dataportal

Automatically exported from code.google.com/p/gbif-dataportal
0 stars 1 forks source link

Enhance the way we collect usage/indexing logs. #49

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
We need to think about several issues to enhance the way we collect our logs, 
and how we provide these logs to our mirrors:

- Filter out or mark logs generated by Googlebot or other search engine bots. 
Google has increased their indexing rate of our data portal 
(data.gbif.org) so our data portal logs have been increasing steadily (more DB 
size). 

- Keep just necessary usage/indexing logs in the montly rollover DB dump given 
to the mirrors, in order to lower the size of the dump file.

Discussions have just started around this issue, so it needs to be discussed 
more to take the best decision.

Original issue reported on code.google.com by josecua...@gmail.com on 17 Aug 2009 at 10:25