search

pombreda / cdpedia

Automatically exported from code.google.com/p/cdpedia
0 stars 0 forks source link

meta: analizar los logs obtenidos con un crawler #148

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Poniendo un crawler a recorrer el minidump se obtienen logs del lado de cdpedia 
y del crawler [0]. Analizandolos se descubren defectos, por ejemplo los de 
issues 146, 147

Con los fixes expuestos en esos issues no salen mas tracebacks en el log de 
cdpedia, pero se pueden analizar para ver si aparecen otros tipos de defectos.

La idea seria marcar en éste issue que secciones son anomalias esperadas y 
cuáles otras son atendidas por otros issues (ver siguientes comentarios) 

Adjunto un script que lee el cdpedia.log ligeramente masajeado (ver comments en 
el codigo para ver como ajustar) y hace algunos análisis basicos.

El reporte que obtengo es:

Statuses detected, with quantity:
200 : 2004
404 : 12990
500 : 2

URLs with 500 status
['/institucional/cc-by-sa.html', '/institucional/wikimedia.html']

Quantity of responses with 404 by prefix
    4
    1 /ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+
    1 /cmp/tutorial
   36 /static/skins/css/images
    4 /static/skins/images/vector/collapsibleNav
   11 /static/skins/images/wikiEditor/dialogs
    8 /static/skins/images/wikiEditor/toolbar
    1 /tutorial
    1 /w
12767 /wiki
    4 /wiki/Copa_Mundial_de_F%C3%BAtbol_de_2006
  128 /wiki/Especial:FuentesDeLibros
    1 /wiki/Lago_Buenos_Aires
    1 /wiki/Lago_O%27Higgins
    1 /wiki/Portal:Astron%C3%A1utica/Video
    1 /wiki/Portal:Astronom%C3%ADa/Actualidad
    1 /wiki/Portal:Astronom%C3%ADa/Destacado
    1 /wiki/Portal:Astronom%C3%ADa/Sab%C3%ADas_que
    1 /wiki/Primera_Divisi%C3%B3n_Venezolana_2001
    1 /wiki/Primera_Divisi%C3%B3n_Venezolana_2006
    1 /wiki/Primera_Divisi%C3%B3n_Venezolana_2008
    1 /wiki/Primera_Divisi%C3%B3n_Venezolana_2012
    1 /wiki/Primera_Divisi%C3%B3n_de_Portugal_2005
    1 /wiki/Primera_Divisi%C3%B3n_de_Portugal_2006
    1 /wiki/Primera_Divisi%C3%B3n_de_Portugal_2007
    1 /wiki/Primera_Divisi%C3%B3n_de_Portugal_2008
    1 /wiki/Primera_Divisi%C3%B3n_de_Portugal_2009
    2 /wiki/S
    1 /wiki/Segunda_Divisi%C3%B3n_Venezolana_2005
    1 /wiki/Segunda_Divisi%C3%B3n_Venezolana_2009
    1 /wiki/Segunda_Divisi%C3%B3n_Venezolana_2011
    1 /wiki/Wikipedia:Tabl%C3%B3n_de_anuncios_de_los_bibliotecarios/Portal/Archivo/Fusi%C3%B3n_de_historiales
    1 /wiki/Wikipedia:Texto_de_la_Licencia_Creative_Commons_Atribuci%C3%B3n
    1 /wiki/Wikipedia:Wikiproyecto
    1 /wiki/Wikiproyecto:Portales

Missing static resources
/static/skins/css/images/add.png
/static/skins/css/images/ajax-loader.gif
/static/skins/css/images/arrow-down-icon.png?1
/static/skins/css/images/audio-icon.png?2
/static/skins/css/images/document-icon.png?2
/static/skins/css/images/feed-icon.png
/static/skins/css/images/file-icon.png?2
/static/skins/css/images/lock-icon.png?2
/static/skins/css/images/mail-icon.png?2
/static/skins/css/images/news-icon.png?2
/static/skins/css/images/page-fade.png?1
/static/skins/css/images/portal-break.png?1
/static/skins/css/images/preferences-base.png?1
/static/skins/css/images/preferences-break.png?1
/static/skins/css/images/preferences-fade.png?1
/static/skins/css/images/remove.png
/static/skins/css/images/search-fade.png?1
/static/skins/css/images/spinner.gif
/static/skins/css/images/tab-break.png?1
/static/skins/css/images/tab-current-fade.png?1
/static/skins/css/images/tab-normal-fade.png?1
/static/skins/css/images/talk-icon.png?2
/static/skins/css/images/ui-bg_flat_0_000000_40x100.png
/static/skins/css/images/ui-bg_flat_100_ffffff_40x100.png
/static/skins/css/images/ui-bg_highlight-hard_100_e2eef6_1x100.png
/static/skins/css/images/ui-bg_highlight-hard_100_ffffff_1x100.png
/static/skins/css/images/ui-bg_highlight-hard_55_fbf9ee_1x100.png
/static/skins/css/images/ui-bg_inset-soft_95_fef1ec_1x100.png
/static/skins/css/images/ui-icons_000000_256x240.png
/static/skins/css/images/ui-icons_2e83ff_256x240.png
/static/skins/css/images/ui-icons_333333_256x240.png
/static/skins/css/images/ui-icons_cd0a0a_256x240.png
/static/skins/css/images/user-icon.png?1
/static/skins/css/images/video-icon.png?2
/static/skins/css/images/watch-icon-loading.gif?1
/static/skins/css/images/watch-icons.png?1
/static/skins/images/vector/collapsibleNav/arrow-down.png?1
/static/skins/images/vector/collapsibleNav/arrow-left.png?1
/static/skins/images/vector/collapsibleNav/arrow-right.png?1
/static/skins/images/vector/collapsibleNav/portal-break.png?1
/static/skins/images/wikiEditor/dialogs/button_disabled.png?1
/static/skins/images/wikiEditor/dialogs/button_down.png?1
/static/skins/images/wikiEditor/dialogs/button_off.png?1
/static/skins/images/wikiEditor/dialogs/button_over.png?1
/static/skins/images/wikiEditor/dialogs/close_x.png?1
/static/skins/images/wikiEditor/dialogs/insert-link-exists.png?1
/static/skins/images/wikiEditor/dialogs/insert-link-external-rtl.png?1
/static/skins/images/wikiEditor/dialogs/insert-link-external.png?1
/static/skins/images/wikiEditor/dialogs/insert-link-invalid.png?1
/static/skins/images/wikiEditor/dialogs/insert-link-notexists.png?1
/static/skins/images/wikiEditor/dialogs/titlebar_fade.png?1
/static/skins/images/wikiEditor/toolbar/arrow-down.png?1
/static/skins/images/wikiEditor/toolbar/arrow-left.png?1
/static/skins/images/wikiEditor/toolbar/arrow-right.png?1
/static/skins/images/wikiEditor/toolbar/base.png?1
/static/skins/images/wikiEditor/toolbar/button-sprite.png?1
/static/skins/images/wikiEditor/toolbar/loading-small.gif?1
/static/skins/images/wikiEditor/toolbar/loading.gif
/static/skins/images/wikiEditor/toolbar/loading.gif?2

[0] http://code.google.com/p/cdpedia/issues/detail?id=143#c4

Original issue reported on code.google.com by ccanepacc@gmail.com on 7 Oct 2012 at 7:50

Attachments:

GoogleCodeExporter commented 9 years ago 
Los dos '500' corresponden a issue 146, que ya tienen un workaround

Original comment by ccanepacc@gmail.com on 7 Oct 2012 at 7:51

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
Creé issue 149 para los '404' que pegan en /static/

Original comment by ccanepacc@gmail.com on 7 Oct 2012 at 8:03

GoogleCodeExporter commented 9 years ago 
Creé issue 150 para '404' con prefix /tutorial

Original comment by ccanepacc@gmail.com on 7 Oct 2012 at 8:58

GoogleCodeExporter commented 9 years ago 
Creé issue 151 para el 404 de cmp/tutorial

Original comment by ccanepacc@gmail.com on 7 Oct 2012 at 9:34

GoogleCodeExporter commented 9 years ago 
Cree issue 152 para el único 404 con prefix /w/

Original comment by ccanepacc@gmail.com on 7 Oct 2012 at 10:16

GoogleCodeExporter commented 9 years ago 
De los 4 404 en el top level uno es /robots.txt que pareceria ignorable; para 
los otros tres creé issue 153

Original comment by ccanepacc@gmail.com on 7 Oct 2012 at 10:53