No dashboards actually show up after running script

[alexanderfichel]

I am using AWS managed service and I get:

$ ./load.sh -url "http://search-abc-123.us-themoon-99.es.amazonaws.com"

{"Message":"User: anonymous is not authorized to perform: es:ESHttpPut on resource: arn:aws:es:us-west-2:1235573471613174:domain/windows-apache-cluster/.kibana/visualization/PgSQL-Errors"}

So it is not using AWS signed requests? Any way you guys could maybe add that feature?

Even if I put open domain policy: I get false for created so nothing gets created...

Loading index pattern filebeat-*:
{"_index":".kibana","_type":"index-pattern","_id":"filebeat-*","_version":3,"created":false}
Loading index pattern packetbeat-*:
{"_index":".kibana","_type":"index-pattern","_id":"packetbeat-*","_version":3,"created":false}
Loading index pattern topbeat-*:
{"_index":".kibana","_type":"index-pattern","_id":"topbeat-*","_version":3,"created":false}
Loading index pattern winlogbeat-*:
{"_index":".kibana","_type":"index-pattern","_id":"winlogbeat-*","_version":3,"created":false}

[tsg]

I marked it as an enhancement request. I'm not too familiar with the AWS search service, is there no way of using basic auth or an HTTP header for auth?

[alexanderfichel]

Well, after I set to open domain policy, the authorization error went away, but all "created" are false

[tsg]

created false just means that the doc was already there and it got overwritten. How do you check if the dashboards are loaded?

[alexanderfichel]

I just go to Kibana dashboards and refresh the browser, shouldn't they just appear there?

[dedemorton]

Did you open the saved dashboards as described in the doc? https://www.elastic.co/guide/en/beats/libbeat/current/view-kibana-dashboards.html

[dedemorton]

To clarify my previous comment: Did you go to the Dashboard page and click the Load Saved Dashboard icon? You need to do that and then select Packetbeat Dashboard from the list, and you'll be able to select from the Beats dashboards.

[alexanderfichel]

Yes, I did that. Actually a similar issue can be seen in stackoverflow:

http://stackoverflow.com/questions/34802545/uploading-dashboards-to-elastic-search-kibana-for-beats-topbeats

[alexanderfichel]

I completely wiped out my indices and ran command:

./load.sh -url "search-123.us-west-2.es.amazonaws.com" -index topbeat-03-30-2016

I get good output:

[root@ip-10-218-249-94 beats-dashboards]# ./load.sh -url "search-new-test-123.us-west-2.es.amazonaws.com" -index topbeat-03-30-2016
Loading dashboards to search-new-test-123.us-west-2.es.amazonaws.com in topbeat-03-30-2016
{"acknowledged":true}{"acknowledged":true}Loading search Cache-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Cache-transactions","_version":1,"created":true}
Loading search DB-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"DB-transactions","_version":1,"created":true}
Loading search Default-Search:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Default-Search","_version":1,"created":true}
Loading search Errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Errors","_version":1,"created":true}
Loading search Filesystem-stats:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Filesystem-stats","_version":1,"created":true}
Loading search HTTP-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"HTTP-errors","_version":1,"created":true}
Loading search MongoDB-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MongoDB-errors","_version":1,"created":true}
Loading search MongoDB-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MongoDB-transactions","_version":1,"created":true}
Loading search MongoDB-transactions-with-write-concern-0:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MongoDB-transactions-with-write-concern-0","_version":1,"created":true}
Loading search MySQL-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MySQL-errors","_version":1,"created":true}
Loading search MySQL-Transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MySQL-Transactions","_version":1,"created":true}
Loading search Packetbeat-Search:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Packetbeat-Search","_version":1,"created":true}
Loading search PgSQL-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"PgSQL-errors","_version":1,"created":true}
Loading search PgSQL-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"PgSQL-transactions","_version":1,"created":true}
Loading search Proc-stats:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Proc-stats","_version":1,"created":true}
Loading search RPC-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"RPC-transactions","_version":1,"created":true}
Loading search System-stats:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"System-stats","_version":1,"created":true}
Loading search Thrift-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Thrift-errors","_version":1,"created":true}
Loading search Thrift-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Thrift-transactions","_version":1,"created":true}
Loading search Web-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Web-transactions","_version":1,"created":true}
Loading search Winlogbeat-Search:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Winlogbeat-Search","_version":1,"created":true}
Loading visualization Cache-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Cache-transactions","_version":1,"created":true}
Loading visualization Client-locations:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Client-locations","_version":1,"created":true}
Loading visualization CPU-usage:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"CPU-usage","_version":1,"created":true}
Loading visualization CPU-usage-per-process:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"CPU-usage-per-process","_version":1,"created":true}
Loading visualization DB-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"DB-transactions","_version":1,"created":true}
Loading visualization Disk-utilization-over-time:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Disk-utilization-over-time","_version":1,"created":true}
Loading visualization Errors-count-over-time:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Errors-count-over-time","_version":1,"created":true}
Loading visualization Errors-vs-successful-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Errors-vs-successful-transactions","_version":1,"created":true}
Loading visualization Event-Levels:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Event-Levels","_version":1,"created":true}
Loading visualization HTTP-codes-for-the-top-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"HTTP-codes-for-the-top-queries","_version":1,"created":true}
Loading visualization HTTP-error-codes-evolution:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"HTTP-error-codes-evolution","_version":1,"created":true}
Loading visualization HTTP-error-codes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"HTTP-error-codes","_version":1,"created":true}
Loading visualization Latency-histogram:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Latency-histogram","_version":1,"created":true}
Loading visualization Levels:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Levels","_version":1,"created":true}
Loading visualization Memory-usage:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Memory-usage","_version":1,"created":true}
Loading visualization Memory-usage-per-process:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Memory-usage-per-process","_version":1,"created":true}
Loading visualization MongoDB-commands:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-commands","_version":1,"created":true}
Loading visualization MongoDB-errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-errors","_version":1,"created":true}
Loading visualization MongoDB-errors-per-collection:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-errors-per-collection","_version":1,"created":true}
Loading visualization MongoDB-in-slash-out-throughput:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-in-slash-out-throughput","_version":1,"created":true}
Loading visualization MongoDB-response-times-and-count:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-response-times-and-count","_version":1,"created":true}
Loading visualization MongoDB-response-times-by-collection:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-response-times-by-collection","_version":1,"created":true}
Loading visualization Most-frequent-MySQL-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Most-frequent-MySQL-queries","_version":1,"created":true}
Loading visualization Most-frequent-PgSQL-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Most-frequent-PgSQL-queries","_version":1,"created":true}
Loading visualization MySQL-Errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MySQL-Errors","_version":1,"created":true}
Loading visualization MySQL-Methods:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MySQL-Methods","_version":1,"created":true}
Loading visualization MySQL-Reads-vs-Writes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MySQL-Reads-vs-Writes","_version":1,"created":true}
Loading visualization Mysql-response-times-percentiles:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Mysql-response-times-percentiles","_version":1,"created":true}
Loading visualization MySQL-throughput:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MySQL-throughput","_version":1,"created":true}
Loading visualization Navigation:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Navigation","_version":1,"created":true}
Loading visualization Number-of-Events:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Number-of-Events","_version":1,"created":true}
Loading visualization Number-of-Events-Over-Time-By-Event-Log:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Number-of-Events-Over-Time-By-Event-Log","_version":1,"created":true}
Loading visualization Number-of-MongoDB-transactions-with-writeConcern-w-equal-0:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Number-of-MongoDB-transactions-with-writeConcern-w-equal-0","_version":1,"created":true}
Loading visualization PgSQL-Errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-Errors","_version":1,"created":true}
Loading visualization PgSQL-Methods:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-Methods","_version":1,"created":true}
Loading visualization PgSQL-Reads-vs-Writes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-Reads-vs-Writes","_version":1,"created":true}
Loading visualization PgSQL-response-times-percentiles:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-response-times-percentiles","_version":1,"created":true}
Loading visualization PgSQL-throughput:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-throughput","_version":1,"created":true}
Loading visualization Process-status:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Process-status","_version":1,"created":true}
Loading visualization Reads-versus-Writes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Reads-versus-Writes","_version":1,"created":true}
Loading visualization Response-times-percentiles:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Response-times-percentiles","_version":1,"created":true}
Loading visualization Response-times-repartition:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Response-times-repartition","_version":1,"created":true}
Loading visualization RPC-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"RPC-transactions","_version":1,"created":true}
Loading visualization Servers:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Servers","_version":1,"created":true}
Loading visualization Slowest-MySQL-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Slowest-MySQL-queries","_version":1,"created":true}
Loading visualization Slowest-PgSQL-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Slowest-PgSQL-queries","_version":1,"created":true}
Loading visualization Slowest-Thrift-RPC-methods:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Slowest-Thrift-RPC-methods","_version":1,"created":true}
Loading visualization Sources:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Sources","_version":1,"created":true}
Loading visualization System-load:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"System-load","_version":1,"created":true}
Loading visualization Thrift-requests-per-minute:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Thrift-requests-per-minute","_version":1,"created":true}
Loading visualization Thrift-response-times-percentiles:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Thrift-response-times-percentiles","_version":1,"created":true}
Loading visualization Thrift-RPC-Errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Thrift-RPC-Errors","_version":1,"created":true}
Loading visualization Top-10-HTTP-requests:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-10-HTTP-requests","_version":1,"created":true}
Loading visualization Top-10-memory-consumers:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-10-memory-consumers","_version":1,"created":true}
Loading visualization Top-10-processes-by-total-CPU-usage:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-10-processes-by-total-CPU-usage","_version":1,"created":true}
Loading visualization Top-Event-IDs:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-Event-IDs","_version":1,"created":true}
Loading visualization Top-processes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-processes","_version":1,"created":true}
Loading visualization Top-slowest-MongoDB-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-slowest-MongoDB-queries","_version":1,"created":true}
Loading visualization Top-Thrift-RPC-calls-with-errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-Thrift-RPC-calls-with-errors","_version":1,"created":true}
Loading visualization Top-Thrift-RPC-methods:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-Thrift-RPC-methods","_version":1,"created":true}
Loading visualization Total-number-of-HTTP-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Total-number-of-HTTP-transactions","_version":1,"created":true}
Loading visualization Total-time-spent-in-each-MongoDB-collection:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Total-time-spent-in-each-MongoDB-collection","_version":1,"created":true}
Loading visualization Web-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Web-transactions","_version":1,"created":true}
Loading dashboard HTTP:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"HTTP","_version":1,"created":true}
Loading dashboard MongoDB-performance:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"MongoDB-performance","_version":1,"created":true}
Loading dashboard MySQL-performance:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"MySQL-performance","_version":1,"created":true}
Loading dashboard Packetbeat-Dashboard:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"Packetbeat-Dashboard","_version":1,"created":true}
Loading dashboard PgSQL-performance:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"PgSQL-performance","_version":1,"created":true}
Loading dashboard Thrift-performance:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"Thrift-performance","_version":1,"created":true}
Loading dashboard Topbeat-Dashboard:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"Topbeat-Dashboard","_version":1,"created":true}
Loading dashboard Winlogbeat-Dashboard:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"Winlogbeat-Dashboard","_version":1,"created":true}
Loading index pattern filebeat-*:
{"_index":"topbeat-03-30-2016","_type":"index-pattern","_id":"filebeat-*","_version":1,"created":true}
Loading index pattern packetbeat-*:
{"_index":"topbeat-03-30-2016","_type":"index-pattern","_id":"packetbeat-*","_version":1,"created":true}
Loading index pattern topbeat-*:
{"_index":"topbeat-03-30-2016","_type":"index-pattern","_id":"topbeat-*","_version":1,"created":true}
Loading index pattern winlogbeat-*:
{"_index":"topbeat-03-30-2016","_type":"index-pattern","_id":"winlogbeat-*","_version":1,"created":true}

Then I ran logstash:

output {
  stdout{ codec => rubydebug}
  amazon_es {
    hosts => ["search-123.us-west-2.es.amazonaws.com"]
    region => "us-west-2"
    aws_access_key_id => 'ABC'
    aws_secret_access_key => 'XYZ'
    index => "topbeat-03-30-2016"
    }
}

Still when I go to kibana, I cannot see the beats dashboards :(

And yes I followed this guide: https://www.elastic.co/guide/en/beats/libbeat/current/view-kibana-dashboards.html

One thing to note, is I never see the predefined index patterns it supposedly creates from the guide above.

[alexanderfichel]

From the AWS console the dashboards actually do get created, but nothing seen in actually kibana dashboard

[monicasarbu]

@GittyHup By default, the load.sh script is loading the Kibana dashboards to the .kibana index. If you want to use .kibana4 instead then you can run the following command:

./load.sh -url "search-123.us-west-2.es.amazonaws.com" -index .kibana4

The topbeat-* index pattern is used by Topbeat to store the system statistics.

[alexanderfichel]

@monicasarbu Thank you!!! I had to change to .kibana-4 instead though. But it finally works!