How to visualize crawl with Kibana?

[rrgirish]

I tried crawling with a couple of sites using the nutch crawler. It shows that it has crawled ~13000 pages. When i click on the visualize button, the kibana dashboard says i have to configure an index pattern.

"logstash-* " doesn't seem to work.

Does this need log.io to work? What sort of values can i give in the index field to see the visualization?

Is there any documentation for this part?

[ahmadia]

Hi @rrgirish!

Yep, this is an issue on our side! No, log.io isn't the issue, and this isn't particularly well documented.

We're using a version of Nutch that pipes processing through Tika and then indexes the results into ElasticSearch. The name of the crawl is then used as the name of the ElasticSearch index. You can look at the local indices with:

curl http://localhost:9200/_cat/indices (Your web browser should find that as well).

It might not be too much work to do the following:

• Synchronize Kibana's configuration every time we create/update a crawl (the crawl model is managed by an SQLite database, Kibana's configuration is stored in ElasticSearch)
• Have the "Visualize" link point to the correct index in Kibana, instead of landing you on Kibana's main page.

[rrgirish]

Thanks!. Yup having the visualize link point to the correct index would be less confusing indeed.

[brittainhard]

@ahmadia how do we get around the problem of having to configure the index in Kibana first? Is there a way just to point kibana directly to an index in elasticsearch without having to go through this step?

We need to figure this out.