problem in creating index patterns

[starlessboi]

Hi, thanks for your great job I have a problem after import of dashboard by load.sh, after import in kibana setting-> indices page on left side there is no [filebeat-]YYYY.MM.DD ... format patterns, the available pattern is filebeat-* ... the output, when importing file by load.sh is as below (everything seem to be normal):

./load.sh -url "http://localhost:9200"ocalhost:9200/.kibana Loading dashboards to http://localhost:9200 in .kibana Loading search Cache-transactions: {"_index":".kibana","_type":"search","_id":"Cache-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search DB-transactions: {"_index":".kibana","_type":"search","_id":"DB-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Default-Search: {"_index":".kibana","_type":"search","_id":"Default-Search","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Errors: {"_index":".kibana","_type":"search","_id":"Errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Filesystem-stats: {"_index":".kibana","_type":"search","_id":"Filesystem-stats","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search HTTP-errors: {"_index":".kibana","_type":"search","_id":"HTTP-errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search MongoDB-errors: {"_index":".kibana","_type":"search","_id":"MongoDB-errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search MongoDB-transactions: {"_index":".kibana","_type":"search","_id":"MongoDB-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search MongoDB-transactions-with-write-concern-0: {"_index":".kibana","_type":"search","_id":"MongoDB-transactions-with-write-concern-0","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search MySQL-errors: {"_index":".kibana","_type":"search","_id":"MySQL-errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search MySQL-Transactions: {"_index":".kibana","_type":"search","_id":"MySQL-Transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Packetbeat-Search: {"_index":".kibana","_type":"search","_id":"Packetbeat-Search","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search PgSQL-errors: {"_index":".kibana","_type":"search","_id":"PgSQL-errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search PgSQL-transactions: {"_index":".kibana","_type":"search","_id":"PgSQL-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Processes: {"_index":".kibana","_type":"search","_id":"Processes","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Proc-stats: {"_index":".kibana","_type":"search","_id":"Proc-stats","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search RPC-transactions: {"_index":".kibana","_type":"search","_id":"RPC-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search System-stats: {"_index":".kibana","_type":"search","_id":"System-stats","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search System-wide: {"_index":".kibana","_type":"search","_id":"System-wide","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Thrift-errors: {"_index":".kibana","_type":"search","_id":"Thrift-errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Thrift-transactions: {"_index":".kibana","_type":"search","_id":"Thrift-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Web-transactions: {"_index":".kibana","_type":"search","_id":"Web-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading search Winlogbeat-Search: {"_index":".kibana","_type":"search","_id":"Winlogbeat-Search","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Average-system-load-across-all-systems: {"_index":".kibana","_type":"visualization","_id":"Average-system-load-across-all-systems","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Cache-transactions: {"_index":".kibana","_type":"visualization","_id":"Cache-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Client-locations: {"_index":".kibana","_type":"visualization","_id":"Client-locations","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization CPU-usage: {"_index":".kibana","_type":"visualization","_id":"CPU-usage","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization CPU-usage-per-process: {"_index":".kibana","_type":"visualization","_id":"CPU-usage-per-process","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization DB-transactions: {"_index":".kibana","_type":"visualization","_id":"DB-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Disk-usage: {"_index":".kibana","_type":"visualization","_id":"Disk-usage","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Disk-usage-overview: {"_index":".kibana","_type":"visualization","_id":"Disk-usage-overview","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Disk-utilization-over-time: {"_index":".kibana","_type":"visualization","_id":"Disk-utilization-over-time","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Errors-count-over-time: {"_index":".kibana","_type":"visualization","_id":"Errors-count-over-time","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Errors-vs-successful-transactions: {"_index":".kibana","_type":"visualization","_id":"Errors-vs-successful-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Event-Levels: {"_index":".kibana","_type":"visualization","_id":"Event-Levels","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Evolution-of-the-CPU-times-per-process: {"_index":".kibana","_type":"visualization","_id":"Evolution-of-the-CPU-times-per-process","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization HTTP-codes-for-the-top-queries: {"_index":".kibana","_type":"visualization","_id":"HTTP-codes-for-the-top-queries","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization HTTP-error-codes-evolution: {"_index":".kibana","_type":"visualization","_id":"HTTP-error-codes-evolution","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization HTTP-error-codes: {"_index":".kibana","_type":"visualization","_id":"HTTP-error-codes","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Latency-histogram: {"_index":".kibana","_type":"visualization","_id":"Latency-histogram","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Levels: {"_index":".kibana","_type":"visualization","_id":"Levels","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Memory-usage: {"_index":".kibana","_type":"visualization","_id":"Memory-usage","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Memory-usage-per-process: {"_index":".kibana","_type":"visualization","_id":"Memory-usage-per-process","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MongoDB-commands: {"_index":".kibana","_type":"visualization","_id":"MongoDB-commands","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MongoDB-errors: {"_index":".kibana","_type":"visualization","_id":"MongoDB-errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MongoDB-errors-per-collection: {"_index":".kibana","_type":"visualization","_id":"MongoDB-errors-per-collection","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MongoDB-in-slash-out-throughput: {"_index":".kibana","_type":"visualization","_id":"MongoDB-in-slash-out-throughput","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MongoDB-response-times-and-count: {"_index":".kibana","_type":"visualization","_id":"MongoDB-response-times-and-count","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MongoDB-response-times-by-collection: {"_index":".kibana","_type":"visualization","_id":"MongoDB-response-times-by-collection","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Most-frequent-MySQL-queries: {"_index":".kibana","_type":"visualization","_id":"Most-frequent-MySQL-queries","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Most-frequent-PgSQL-queries: {"_index":".kibana","_type":"visualization","_id":"Most-frequent-PgSQL-queries","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MySQL-Errors: {"_index":".kibana","_type":"visualization","_id":"MySQL-Errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MySQL-Methods: {"_index":".kibana","_type":"visualization","_id":"MySQL-Methods","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MySQL-Reads-vs-Writes: {"_index":".kibana","_type":"visualization","_id":"MySQL-Reads-vs-Writes","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Mysql-response-times-percentiles: {"_index":".kibana","_type":"visualization","_id":"Mysql-response-times-percentiles","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization MySQL-throughput: {"_index":".kibana","_type":"visualization","_id":"MySQL-throughput","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Navigation: {"_index":".kibana","_type":"visualization","_id":"Navigation","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Number-of-Events: {"_index":".kibana","_type":"visualization","_id":"Number-of-Events","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Number-of-Events-Over-Time-By-Event-Log: {"_index":".kibana","_type":"visualization","_id":"Number-of-Events-Over-Time-By-Event-Log","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Number-of-MongoDB-transactions-with-writeConcern-w-equal-0: {"_index":".kibana","_type":"visualization","_id":"Number-of-MongoDB-transactions-with-writeConcern-w-equal-0","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization PgSQL-Errors: {"_index":".kibana","_type":"visualization","_id":"PgSQL-Errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization PgSQL-Methods: {"_index":".kibana","_type":"visualization","_id":"PgSQL-Methods","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization PgSQL-Reads-vs-Writes: {"_index":".kibana","_type":"visualization","_id":"PgSQL-Reads-vs-Writes","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization PgSQL-response-times-percentiles: {"_index":".kibana","_type":"visualization","_id":"PgSQL-response-times-percentiles","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization PgSQL-throughput: {"_index":".kibana","_type":"visualization","_id":"PgSQL-throughput","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Process-status: {"_index":".kibana","_type":"visualization","_id":"Process-status","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Reads-versus-Writes: {"_index":".kibana","_type":"visualization","_id":"Reads-versus-Writes","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Response-times-percentiles: {"_index":".kibana","_type":"visualization","_id":"Response-times-percentiles","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Response-times-repartition: {"_index":".kibana","_type":"visualization","_id":"Response-times-repartition","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization RPC-transactions: {"_index":".kibana","_type":"visualization","_id":"RPC-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Servers: {"_index":".kibana","_type":"visualization","_id":"Servers","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Slowest-MySQL-queries: {"_index":".kibana","_type":"visualization","_id":"Slowest-MySQL-queries","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Slowest-PgSQL-queries: {"_index":".kibana","_type":"visualization","_id":"Slowest-PgSQL-queries","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Slowest-Thrift-RPC-methods: {"_index":".kibana","_type":"visualization","_id":"Slowest-Thrift-RPC-methods","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Sources: {"_index":".kibana","_type":"visualization","_id":"Sources","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization System-load: {"_index":".kibana","_type":"visualization","_id":"System-load","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Thrift-requests-per-minute: {"_index":".kibana","_type":"visualization","_id":"Thrift-requests-per-minute","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Thrift-response-times-percentiles: {"_index":".kibana","_type":"visualization","_id":"Thrift-response-times-percentiles","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Thrift-RPC-Errors: {"_index":".kibana","_type":"visualization","_id":"Thrift-RPC-Errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Top-10-HTTP-requests: {"_index":".kibana","_type":"visualization","_id":"Top-10-HTTP-requests","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Top-10-memory-consumers: {"_index":".kibana","_type":"visualization","_id":"Top-10-memory-consumers","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Top-10-processes-by-total-CPU-usage: {"_index":".kibana","_type":"visualization","_id":"Top-10-processes-by-total-CPU-usage","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Top-Event-IDs: {"_index":".kibana","_type":"visualization","_id":"Top-Event-IDs","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Top-processes: {"_index":".kibana","_type":"visualization","_id":"Top-processes","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Top-slowest-MongoDB-queries: {"_index":".kibana","_type":"visualization","_id":"Top-slowest-MongoDB-queries","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Top-Thrift-RPC-calls-with-errors: {"_index":".kibana","_type":"visualization","_id":"Top-Thrift-RPC-calls-with-errors","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Top-Thrift-RPC-methods: {"_index":".kibana","_type":"visualization","_id":"Top-Thrift-RPC-methods","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Total-number-of-HTTP-transactions: {"_index":".kibana","_type":"visualization","_id":"Total-number-of-HTTP-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Total-time-spent-in-each-MongoDB-collection: {"_index":".kibana","_type":"visualization","_id":"Total-time-spent-in-each-MongoDB-collection","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading visualization Web-transactions: {"_index":".kibana","_type":"visualization","_id":"Web-transactions","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading dashboard HTTP: {"_index":".kibana","_type":"dashboard","_id":"HTTP","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading dashboard MongoDB-performance: {"_index":".kibana","_type":"dashboard","_id":"MongoDB-performance","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading dashboard MySQL-performance: {"_index":".kibana","_type":"dashboard","_id":"MySQL-performance","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading dashboard Packetbeat-Dashboard: {"_index":".kibana","_type":"dashboard","_id":"Packetbeat-Dashboard","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading dashboard PgSQL-performance: {"_index":".kibana","_type":"dashboard","_id":"PgSQL-performance","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading dashboard Thrift-performance: {"_index":".kibana","_type":"dashboard","_id":"Thrift-performance","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading dashboard Topbeat-Dashboard: {"_index":".kibana","_type":"dashboard","_id":"Topbeat-Dashboard","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true} Loading dashboard Winlogbeat-Dashboard: {"_index":".kibana","_type":"dashboard","_id":"Winlogbeat-Dashboard","_version":1,"shards":{"total":2,"successful":1,"failed":0},"created":true} Loading index pattern filebeat-: {"_index":".kibana","_type":"index-pattern","id":"filebeat-","_version":1,"shards":{"total":2,"successful":1,"failed":0},"created":true} Loading index pattern packetbeat-: {"_index":".kibana","_type":"index-pattern","id":"packetbeat-","_version":1,"shards":{"total":2,"successful":1,"failed":0},"created":true} Loading index pattern topbeat-: {"_index":".kibana","_type":"index-pattern","id":"topbeat-","_version":1,"shards":{"total":2,"successful":1,"failed":0},"created":true} Loading index pattern winlogbeat-: {"_index":".kibana","_type":"index-pattern","id":"winlogbeat-","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

please tell me how can I fix this.

[borian]

I have also experienced problems when creating an index The load.sh script parses the name wrong and adds an extra comma at the end. filebeat-*, instead of filebeat-*

[ruflin]

@starlessboi In Kibana for you need to select Use event times to create index names [DEPRECATED] do set daily patterns. We need to update our docs here. Thanks for bringing this up. It is ok here to use ?filebeat-* as pattern. Is that an issue for you?

@borian Which OS are you using?

@dedemorton FYI

[borian]

@ruflin I use a docker container with debian 7

[borian]

OK, I found the source of my problem, when "title": "topbeat-*" is not the last element in the json document the awk command also reads the json , delimiter

I was using my custom file, and did not define it as the last json element

[ruflin]

@borian Thanks for tracking this done. We should probably fix that in the import script?

Update: Just saw https://github.com/elastic/beats-dashboards/pull/92

[monicasarbu]

@starlessboi The format [filebeat-]YYYY.MM.DD is obsolete in Kibana and I replaced it recently with the index pattern filebeat-*. Only the name changed, the format of the index pattern is the same.

[monicasarbu]

Thank you @borian for the quick fix.

[monicasarbu]

Closing it as the docs are also updated in the current version. @starlessboi feel free to re-open it if you still encounter problems.