search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 324 forks source link

ELK - export.json for Kibana contains incorrect field (if log is updated via filebeat) #74

Closed Paw2Pom closed 8 years ago

Paw2Pom commented 8 years ago

Based on the scrapy-cluster doc page - ELK, I have configured to use filebeat to transfer log info directly into elasticsearch (i.e. no logstash). An elasticsearch index is created, however when I try to import the "export.json" via Kibana to display the dashboard, I have encountered the following error:

Saved Objects: Could not locate that index-pattern-field (id: timestamp)

I thought its because logstash used the naming of "timestamp" instead of "@timestamp" by filebeat, I have updated the "export.json" so that all "timestamp" is replaced with "@timestamp". Than I have encountered another error:

Saved Objects: Could not locate that index-pattern-field (id: status_code)

I am not quite sure how to solve this one. Any advice will be appreciated.

Below are the configurations I have used:

Configuration file for filebeat : filebeat.yml

filebeat:

List of prospectors to fetch data.

prospectors:

  paths:
    - /var/log/scrapy-cluster/*.log
  input_type: log
  document_type: log_scrapy-cluster
  ignore_older: 5m
  scan_frequency: 5s

Name of the registry file. Per default it is put in the current working

directory. In case the working directory is changed after when running

filebeat again, indexing starts from the beginning again.

registry_file: /var/lib/filebeat/registry

output: elasticsearch: hosts: ["elasticsearch:9200"] index: "logs-scrapy-cluster" template: name: "logs-*"

name: "logs-scrapy-cluster"

  path: "/opt/filebeat/template_sc.json"
  overwrite: true

shipper: name: filebeat-sc tags: ["scrapy-cluster"]

logging: to_files: true files: path: /opt/filebeat/log name: filebeat rotateeverybytes: 10485760 # = 10MB keepfiles: 2 level: error

Template file for elasticsearch : template_.json

{ "template" : "logs-_", "order" : 0, "settings" : { "index.refresh_interval" : "5s" }, "mappings" : { "default" : { "dynamic_templates" : [ { "message_field" : { "mapping" : { "index" : "not_analyzed", "omit_norms" : true, "include_in_parent": true, "type" : "string" }, "match_mapping_type" : "string", "match" : "message" } }, { "string_fields" : { "mapping" : { "index" : "not_analyzed", "omit_norms" : true, "include_in_parent": true, "type" : "string" }, "match_mappingtype" : "string", "match" : "" } } ], "properties" : { "@version" : { "index" : "not_analyzed", "type" : "string" } }, "_all" : { "enabled" : true } } }, "aliases" : { } }

Import object for Kibana dashboard : export_v1_timestamp.json

[ { "_id": "Scrapy-Cluster-Overall", "_type": "dashboard", "_source": { "title": "Scrapy Cluster Overall", "hits": 0, "description": "", "panelsJSON": "[{\"col\":1,\"id\":\"Appid-Pie-Chart\",\"panelIndex\":1,\"row\":3,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"col\":1,\"id\":\"Crawler-Machine-Load\",\"panelIndex\":2,\"row\":7,\"size_x\":3,\"size_y\":3,\"type\":\"visualization\"},{\"col\":4,\"id\":\"Crawler-Response-Codes\",\"panelIndex\":3,\"row\":7,\"size_x\":9,\"size_y\":3,\"type\":\"visualization\"},{\"col\":4,\"id\":\"Kafka-Monitor-Histogram\",\"panelIndex\":4,\"row\":1,\"size_x\":9,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"Scrapy-Cluster-Navigation\",\"panelIndex\":5,\"row\":1,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"col\":4,\"id\":\"Crawler-Backlog\",\"panelIndex\":6,\"row\":4,\"size_x\":9,\"size_y\":3,\"type\":\"visualization\"},{\"id\":\"Crawler-Spider-Count\",\"type\":\"visualization\",\"panelIndex\":7,\"size_x\":3,\"size_y\":2,\"col\":1,\"row\":5}]", "optionsJSON": "{\"darkTheme\":false}", "uiStateJSON": "{\"P-1\":{\"vis\":{\"legendOpen\":false}},\"P-2\":{\"vis\":{\"legendOpen\":false}}}", "version": 1, "timeRestore": false, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"analyzewildcard\":true,\"query\":\"\"}}}]}" } } }, { "_id": "Scrapy-Cluster-Kafka-Monitor", "_type": "dashboard", "_source": { "title": "Scrapy Cluster Kafka Monitor", "hits": 0, "description": "", "panelsJSON": "[{\"col\":1,\"id\":\"Appid-Pie-Chart\",\"panelIndex\":1,\"row\":5,\"size_x\":4,\"size_y\":4,\"type\":\"visualization\"},{\"col\":4,\"id\":\"Kafka-Monitor-Histogram\",\"panelIndex\":3,\"row\":1,\"size_x\":9,\"size_y\":4,\"type\":\"visualization\"},{\"col\":1,\"id\":\"Kafka-Monitor-total-lifetime\",\"panelIndex\":4,\"row\":3,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"col\":1,\"columns\":[\"level\",\"appid\",\"crawlid\",\"message\",\"priority\",\"spiderid\",\"parsed\",\"valid\"],\"id\":\"Scrapy-Cluster-Kafka-Monitor\",\"panelIndex\":5,\"row\":9,\"size_x\":12,\"size_y\":4,\"sort\":[\"timestamp\",\"desc\"],\"type\":\"search\"},{\"col\":1,\"id\":\"Scrapy-Cluster-Navigation\",\"panelIndex\":6,\"row\":1,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"id\":\"Kafka-Monitor-Stats-Fail\",\"type\":\"visualization\",\"panelIndex\":7,\"size_x\":8,\"size_y\":2,\"col\":5,\"row\":7},{\"id\":\"Kafka-Monitor-Stats-Total\",\"type\":\"visualization\",\"panelIndex\":8,\"size_x\":8,\"size_y\":2,\"col\":5,\"row\":5}]", "optionsJSON": "{\"darkTheme\":false}", "uiStateJSON": "{\"P-1\":{\"vis\":{\"legendOpen\":false}}}", "version": 1, "timeRestore": false, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"analyzewildcard\":true,\"query\":\"\"}}}]}" } } }, { "_id": "Scrapy-Cluster-Crawler", "_type": "dashboard", "_source": { "title": "Scrapy Cluster Crawler", "hits": 0, "description": "", "panelsJSON": "[{\"col\":5,\"id\":\"Crawler-Host-Percent\",\"panelIndex\":1,\"row\":7,\"size_x\":8,\"size_y\":3,\"type\":\"visualization\"},{\"col\":5,\"id\":\"Crawler-Response-Codes\",\"panelIndex\":2,\"row\":1,\"size_x\":8,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"columns\":[\"level\",\"message\",\"appid\",\"crawlid\",\"host\",\"response_url\",\"status_code\"],\"id\":\"Scrapy-Cluster-Crawler\",\"panelIndex\":3,\"row\":13,\"size_x\":12,\"size_y\":4,\"sort\":[\"timestamp\",\"desc\"],\"type\":\"search\"},{\"col\":1,\"id\":\"Crawler-Appid-Crawlid\",\"panelIndex\":4,\"row\":5,\"size_x\":4,\"size_y\":2,\"type\":\"visualization\"},{\"col\":5,\"id\":\"Crawler-Kafka-Success\",\"panelIndex\":5,\"row\":4,\"size_x\":8,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"Scrapy-Cluster-Navigation\",\"panelIndex\":6,\"row\":1,\"size_x\":2,\"size_y\":2,\"type\":\"visualization\"},{\"col\":3,\"id\":\"Crawler-Machine-Load\",\"panelIndex\":7,\"row\":1,\"size_x\":2,\"size_y\":2,\"type\":\"visualization\"},{\"col\":1,\"id\":\"Crawler-Status-Codes\",\"panelIndex\":8,\"row\":7,\"size_x\":4,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"Crawler-Spider-Count\",\"panelIndex\":9,\"row\":3,\"size_x\":4,\"size_y\":2,\"type\":\"visualization\"},{\"id\":\"Link-Spider-Stats\",\"type\":\"visualization\",\"panelIndex\":10,\"size_x\":12,\"size_y\":3,\"col\":1,\"row\":10}]", "optionsJSON": "{\"darkTheme\":false}", "uiStateJSON": "{\"P-1\":{\"vis\":{\"legendOpen\":false}},\"P-4\":{\"vis\":{\"legendOpen\":false}},\"P-7\":{\"vis\":{\"legendOpen\":false}},\"P-8\":{\"vis\":{\"legendOpen\":false}}}", "version": 1, "timeRestore": false, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"analyzewildcard\":true,\"query\":\"\"}}}]}" } } }, { "_id": "Scrapy-Cluster-Redis-Monitor", "_type": "dashboard", "_source": { "title": "Scrapy Cluster Redis Monitor", "hits": 0, "description": "", "panelsJSON": "[{\"col\":1,\"id\":\"Scrapy-Cluster-Navigation\",\"panelIndex\":3,\"row\":1,\"size_x\":2,\"size_y\":2,\"type\":\"visualization\"},{\"col\":1,\"columns\":[\"level\",\"message\",\"total_backlog\",\"total_spider_count\",\"queue_link.spider_backlog\",\"queue_wandering.spider_backlog\"],\"id\":\"Scrapy-Cluster-Redis-Monitor\",\"panelIndex\":4,\"row\":12,\"size_x\":12,\"size_y\":5,\"sort\":[\"timestamp\",\"desc\"],\"type\":\"search\"},{\"col\":3,\"id\":\"Unique-Spider-Count\",\"panelIndex\":5,\"row\":1,\"size_x\":2,\"size_y\":2,\"type\":\"visualization\"},{\"col\":5,\"id\":\"Crawler-Backlog\",\"panelIndex\":6,\"row\":1,\"size_x\":8,\"size_y\":5,\"type\":\"visualization\"},{\"col\":1,\"id\":\"Redis-Monitor-Appid\",\"panelIndex\":7,\"row\":3,\"size_x\":4,\"size_y\":3,\"type\":\"visualization\"},{\"col\":5,\"id\":\"Redis-Monitor-Stats-Fail\",\"panelIndex\":8,\"row\":6,\"size_x\":8,\"size_y\":3,\"type\":\"visualization\"},{\"col\":5,\"id\":\"Redis-Monitor-Stats-Total\",\"panelIndex\":9,\"row\":9,\"size_x\":8,\"size_y\":3,\"type\":\"visualization\"},{\"id\":\"Redis-Monitor-Messages\",\"type\":\"visualization\",\"panelIndex\":10,\"size_x\":4,\"size_y\":6,\"col\":1,\"row\":6}]", "optionsJSON": "{\"darkTheme\":false}", "uiStateJSON": "{\"P-7\":{\"vis\":{\"legendOpen\":false}}}", "version": 1, "timeRestore": false, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"analyzewildcard\":true,\"query\":\"\"}}}]}" } } }, { "_id": "Scrapy-Cluster-Kafka-Monitor", "_type": "search", "source": { "title": "Scrapy Cluster Kafka Monitor", "description": "", "hits": 0, "columns": [ "level", "appid", "crawlid", "message", "priority", "spiderid", "parsed", "valid" ], "sort": [ "timestamp", "desc" ], "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"index\":\"logs-\",\"query\":{\"query_string\":{\"query\":\"logger:sc-kafka-monitor\",\"analyze_wildcard\":true}},\"filter\":[],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"posttags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"\":{}},\"require_field_match\":false,\"fragment_size\":2147483647}}" } } }, { "_id": "Scrapy-Cluster-Crawler", "_type": "search", "_source": { "title": "Scrapy Cluster Crawler", "description": "", "hits": 0, "columns": [ "level", "message", "appid", "crawlid", "host", "response_url", "statuscode" ], "sort": [ "timestamp", "desc" ], "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"index\":\"logs-\",\"query\":{\"query_string\":{\"query\":\"logger:sc-crawler\",\"analyze_wildcard\":true}},\"filter\":[],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"posttags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"\":{}},\"require_field_match\":false,\"fragment_size\":2147483647}}" } } }, { "_id": "Scrapy-Cluster-Redis-Monitor", "_type": "search", "_source": { "title": "Scrapy Cluster Redis Monitor", "description": "", "hits": 0, "columns": [ "level", "message", "total_backlog", "total_spider_count", "queue_link.spider_backlog", "queue_wandering.spiderbacklog" ], "sort": [ "timestamp", "desc" ], "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"index\":\"logs-\",\"query\":{\"query_string\":{\"query\":\"logger:sc-redis-monitor\",\"analyze_wildcard\":true}},\"filter\":[],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"posttags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"\":{}},\"require_field_match\":false,\"fragment_size\":2147483647}}" } } }, { "_id": "Crawler-Response-Codes", "_type": "visualization", "_source": { "title": "Crawler Response Codes", "visState": "{\"title\":\"New Visualization\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"status_code\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", "uiStateJSON": "{\"vis\":{\"colors\":{\"200\":\"#7EB26D\"}}}", "description": "", "savedSearchId": "Scrapy-Cluster-Crawler", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Crawler-Host-Percent", "_type": "visualization", "_source": { "title": "Crawler Host Percent", "visState": "{\"title\":\"New Visualization\",\"type\":\"area\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"smoothLines\":true,\"scale\":\"linear\",\"interpolate\":\"linear\",\"mode\":\"percentage\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"host\",\"size\":20,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Crawler", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Kafka-Monitor-total-lifetime", "_type": "visualization", "_source": { "title": "Kafka Monitor total lifetime", "visState": "{\"title\":\"New Visualization\",\"type\":\"metric\",\"params\":{\"fontSize\":60},\"aggs\":[{\"id\":\"2\",\"type\":\"max\",\"schema\":\"metric\",\"params\":{\"field\":\"total_lifetime\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Kafka-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Crawler-Kafka-Success", "_type": "visualization", "_source": { "title": "Crawler Kafka Success", "visState": "{\"title\":\"New Visualization\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"success\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Crawler", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Appid-Pie-Chart", "_type": "visualization", "_source": { "title": "Appid Pie Chart", "visState": "{\"title\":\"Appid Pie Chart\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"isDonut\":true},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"filters\",\"schema\":\"segment\",\"params\":{\"filters\":[{\"input\":{\"query\":{\"query_string\":{\"query\":\"parsed:true AND valid:true\",\"analyze_wildcard\":true}}},\"label\":\"\"},{\"input\":{\"query\":{\"query_string\":{\"query\":\"parsed:true AND valid:false\",\"analyze_wildcard\":true}}}},{\"input\":{\"query\":{\"query_string\":{\"query\":\"parsed:false AND valid:false\",\"analyze_wildcard\":true}}}}]}},{\"id\":\"3\",\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"appid\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Kafka-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Crawler-Appid-Crawlid", "_type": "visualization", "_source": { "title": "Crawler Appid Crawlid", "visState": "{\"title\":\"New Visualization\",\"type\":\"pie\",\"params\":{\"addLegend\":true,\"addTooltip\":true,\"isDonut\":false,\"shareYAxis\":true},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"appid\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}},{\"id\":\"3\",\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"crawlid\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Crawler", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Crawler-Machine-Load", "_type": "visualization", "_source": { "title": "Crawler Machine Load", "visState": "{\"title\":\"New Visualization\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"host\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Crawler", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Crawler-Status-Codes", "_type": "visualization", "_source": { "title": "Crawler Status Codes", "visState": "{\"title\":\"New Visualization\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"isDonut\":true},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"status_code\",\"size\":25,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Crawler", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Scrapy-Cluster-Navigation", "_type": "visualization", "_source": { "title": "Scrapy Cluster Navigation", "visState": "{\"title\":\"Scrapy Cluster Navigation\",\"type\":\"markdown\",\"params\":{\"markdown\":\"Overall\n\nCrawler\n\nKafka Monitor\n\nRedis Monitor\"},\"aggs\":[],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" } } }, { "_id": "Kafka-Monitor-Histogram", "_type": "visualization", "_source": { "title": "Kafka Monitor Histogram", "visState": "{\"title\":\"Kafka Monitor Histogram\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"filters\",\"schema\":\"group\",\"params\":{\"filters\":[{\"input\":{\"query\":{\"query_string\":{\"query\":\"valid:true\",\"analyze_wildcard\":true}}},\"label\":\"\"},{\"input\":{\"query\":{\"query_string\":{\"query\":\"valid:false\",\"analyze_wildcard\":true}}}}]}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Kafka-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Crawler-Spider-Count", "_type": "visualization", "_source": { "title": "Crawler Spider Count", "visState": "{\"aggs\":[{\"id\":\"1\",\"params\":{\"field\":\"total_spider_count\"},\"schema\":\"metric\",\"type\":\"avg\"}],\"listeners\":{},\"params\":{\"fontSize\":60},\"title\":\"New Visualization\",\"type\":\"metric\"}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Redis-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Unique-Spider-Count", "_type": "visualization", "_source": { "title": "Unique Spider Count", "visState": "{\"title\":\"New Visualization\",\"type\":\"metric\",\"params\":{\"fontSize\":60},\"aggs\":[{\"id\":\"1\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"unique_spider_count\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Redis-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Redis-Monitor-Stats-Total", "_type": "visualization", "_source": { "title": "Redis Monitor Stats Total", "visState": "{\"title\":\"New Visualization\",\"type\":\"line\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"showCircles\":true,\"smoothLines\":true,\"interpolate\":\"linear\",\"scale\":\"square root\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_900\"}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_3600\"}},{\"id\":\"4\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_21600\"}},{\"id\":\"5\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_43200\"}},{\"id\":\"6\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_86400\"}},{\"id\":\"7\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_604800\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Redis-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Kafka-Monitor-Stats-Total", "_type": "visualization", "_source": { "title": "Kafka Monitor Stats Total", "visState": "{\"title\":\"Kafka Monitor Stats Total\",\"type\":\"line\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"showCircles\":true,\"smoothLines\":true,\"interpolate\":\"linear\",\"scale\":\"square root\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_900\"}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_3600\"}},{\"id\":\"4\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_21600\"}},{\"id\":\"5\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_43200\"}},{\"id\":\"6\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_86400\"}},{\"id\":\"7\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"total_604800\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Kafka-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Kafka-Monitor-Stats-Fail", "_type": "visualization", "_source": { "title": "Kafka Monitor Stats Fail", "visState": "{\"title\":\"Kafka Monitor Stats Fail\",\"type\":\"line\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"showCircles\":true,\"smoothLines\":true,\"interpolate\":\"linear\",\"scale\":\"square root\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_900\"}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_3600\"}},{\"id\":\"4\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_21600\"}},{\"id\":\"5\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_43200\"}},{\"id\":\"6\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_86400\"}},{\"id\":\"7\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_604800\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Kafka-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Redis-Monitor-Stats-Fail", "_type": "visualization", "_source": { "title": "Redis Monitor Stats Fail", "visState": "{\"title\":\"Redis Monitor Stats Fail\",\"type\":\"line\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"showCircles\":true,\"smoothLines\":true,\"interpolate\":\"linear\",\"scale\":\"square root\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_900\"}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_3600\"}},{\"id\":\"4\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_21600\"}},{\"id\":\"5\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_43200\"}},{\"id\":\"6\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_86400\"}},{\"id\":\"7\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"fail_604800\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Redis-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Crawler-Backlog", "_type": "visualization", "_source": { "title": "Crawler Backlog", "visState": "{\"title\":\"Redis Spider Backlog\",\"type\":\"line\",\"params\":{\"addLegend\":true,\"addTimeMarker\":false,\"addTooltip\":true,\"defaultYExtents\":false,\"drawLinesBetweenPoints\":true,\"interpolate\":\"linear\",\"radiusRatio\":\"7\",\"scale\":\"square root\",\"setYExtents\":false,\"shareYAxis\":true,\"showCircles\":true,\"smoothLines\":false,\"times\":[],\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"queue_link.spider_backlog\"}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"queue_wandering.spider_backlog\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Redis-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Link-Spider-Stats", "_type": "visualization", "_source": { "title": "Link Spider Stats", "visState": "{\"title\":\"New Visualization\",\"type\":\"line\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"showCircles\":true,\"smoothLines\":false,\"interpolate\":\"linear\",\"scale\":\"square root\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"link_200_900\"}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"link_403_900\"}},{\"id\":\"4\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"link_404_900\"}},{\"id\":\"5\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"link_200_3600\"}},{\"id\":\"6\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"link_403_3600\"}},{\"id\":\"7\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"link_404_3600\"}},{\"id\":\"8\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"link_200_86400\"}},{\"id\":\"9\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"link_403_86400\"}},{\"id\":\"10\",\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"link_404_86400\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Redis-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Redis-Monitor-Appid", "_type": "visualization", "_source": { "title": "Redis Monitor Appid", "visState": "{\"title\":\"New Visualization\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"appid\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Redis-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } }, { "_id": "Redis-Monitor-Messages", "_type": "visualization", "_source": { "title": "Redis Monitor Messages", "visState": "{\"title\":\"New Visualization\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"message\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", "uiStateJSON": "{}", "description": "", "savedSearchId": "Scrapy-Cluster-Redis-Monitor", "version": 1, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"filter\":[]}" } } } ]

madisonb commented 8 years ago

@timestamp is the Logstash generated timestamp and while you can use that time field I don't recommend it (since the time that the log was written could be different than the time that the log was sent to Elasticsearch. timestamp is written in the log by the logger, whereas @timestamp is via Logstash.

I had added a couple new notes of documentation on the dev branch (where all active work is done) that I think can explain your problem. The issue is that Kibana will not import a visualization of a mapping it does not have yet. This is typical with a brand new ELK stack that has a handful of logs but not the whole thing.

My recommendation is that before you click "Import" that you actually use your cluster and ensure the index pattern has ~130 different fields known before you import the export.json. This is noted in the first blue note box here. You can issue the following command python kafka_monitor.py feed '{"url":"cnn.com", "appid":"testApp", "crawlid":"abc123", "maxdepth":2}' and in past experience that will generate all of the required log fields that you need after about 5 mins worth of crawling.

Also to note, you are free to use your own Kibana dashboards and visualizations, the hardest part is getting the log data into Elasticsearch.

Paw2Pom commented 8 years ago

After testing both filebeat and logstash, and pre-crawl the cnn.com before I setup the Kibana, i found out that the approach specified in the documentation will only work for logstash setup (i.e. /var/log/scrapy-cluster -> logstash -> elasticsearch -> kibana).

If using filebeat, with the above configuration I provided earlier, there will only be around 10-ish fields appearing, (and does not have the option of "timestamp" but "@timestamp" to select from) which include:

_index string
offset number
count number
input_type string
source string
message string
type string
tags string
beat.name string
beat.hostname string
@timestamp date
@version string
_source _source
_id string
_type string
_score number

If using logstash, yes, there are about 130-ish fields appearing. However when loading up the export.json file, there are still error occurring, such as:

Saved Objects: Could not locate that index-pattern-field (id: queue_wandering.spider_backlog)

Also, not all dashboards are working accordingly, and below are the attached screenshots of dashboards:

[Dashboard - Overall] screen shot 2016-09-07 at 1 31 14 pm

[Dashboard - Crawler] screen shot 2016-09-07 at 1 32 12 pm screen shot 2016-09-07 at 1 32 20 pm

[Dashboard - Kafka-Monitor] screen shot 2016-09-07 at 1 33 36 pm

[Dashboard - Redis-Monitor] screen shot 2016-09-07 at 1 34 04 pm

But if by looking at elasticsearch cluster, it seems the files are entering correctly: screen shot 2016-09-07 at 1 35 43 pm screen shot 2016-09-07 at 1 36 08 pm

madisonb commented 8 years ago

I only see 3 visualizations that are not working in your screenshots above, and I would triple check that your index pattern is refreshed within Kibana so that it knows certain fields exist. Thanks for verifying that indeed the logstash -> kibana setup works, as that is what is outlined in the documentation. I have not used FileBeat but it appears that you still need to ship your logs from FileBeat to Logstash in order to be processed.

The following works for me: (note I am on the dev branch where Docker is setup)

$ cd elk
$ docker-compose -f docker-compose.elk.yml up -d
# Note at this point, I can hit Kibana but it has no index pattern
$ docker exec -it elk_kafka_monitor_1 bash
$ python kafka_monitor.py feed '{"url":"cnn.com", "appid":"test", "crawlid":"abc1234",
"maxdepth":1}'

I would like to close this because: a) It is not an issue with the actual code running the scraping cluster b) It is a configuration issue with Kibana or your logstash setup. I would seek help within their documentation, as this isnt the place to get support for Kibana problems c) I cannot reproduce, given the steps I have outlined both in the documentation here and here, but also in the steps outlined in prior comments.

I understand there may be some quirks, because Kibana doesnt really 100% support what we are trying to do here, but if you get get the logs into Elasticsearch (which I see you already have) then you can always create your own visualizations that fit your needs. I would start with a Vertical Bar Char based on a Date Histogram as a good starting point.

Paw2Pom commented 8 years ago

Yes, once I have switched to dev branch, it works! Much thanks Madison! :D