search

cedardevs / onestop

OneStop is a data discovery system being built by CIRES researchers on a grant from the NOAA National Centers for Environmental Information. We welcome contributions from the community!
GNU General Public License v2.0
44 stars 20 forks source link

Automatically load kibana dashboards #1297

Open erinreeves opened 4 years ago

erinreeves commented 4 years ago

Summary

As a metadata manager
I want to have dashboards loaded automatically in kibana
So that I can use the dashboards

Tasks

erinreeves commented 4 years ago

Few options: If we loaded the kibana dashboard via the API we'd still need a script then to automatically get kicked off, and expose that port (which seems a bit much simply for loading initial data). Oher thought is to copy the file over to the container and you could either do the above but do the script from within the container or have kibana configured to automatically load the file when kibana is ready to do so.

Evan: have kibana configured to automatically load the file when kibana is ready to do so my guess is that that’s your easiest path based on where it sounds like you are now

erinreeves commented 4 years ago

created branch 1297-kibana-dashboards

erinreeves commented 4 years ago

Kibana info so as to load the dashboard json file.

  1. we running ECK WHICH IS AN OPERATOR
  2. its going to be different from the conventional one (example under helm/onestop-registry)
erinreeves commented 4 years ago

To accomplish this would need to upload a file via kubernetes to the kibana container and put the path to this file in the metricbeat.yaml under the sections with the dashboards enabled section:

  dashboards:
    enabled: true
    file:

or do url:

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-dashboards.html

erinreeves commented 4 years ago

I seem to not have some containers up, one of them being the filebeat one which is where the config adjustment to turn on dashboards auto loading is

NAME READY STATUS RESTARTS AGE onestop-client-cbfbc4668-vrzqn 1/1 Running 0 15h onestop-dev-cp-control-center-7cc7cb97d8-8h692 1/1 Running 0 15h onestop-dev-cp-kafka-0 2/2 Running 0 15h onestop-dev-cp-schema-registry-6b8859c769-8r8hq 2/2 Running 2 15h onestop-dev-cp-zookeeper-0 2/2 Running 0 15h onestop-dev-es-default-0 1/1 Running 0 15h onestop-dev-ingress-nginx-controller-5f9c9d7569-9ppkz 1/1 Running 0 15h onestop-dev-kb-6c9f7c8bdb-xzm5n 1/1 Running 0 15h onestop-indexer-779c5b57cf-89sqt 2/2 Running 4 15h onestop-manager-794cbb5d8c-rq4r2 2/2 Running 1 15h onestop-registry-0 2/2 Running 5 15h onestop-search-6564c8b9d-t8dsj 2/2 Running 2 15h

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-dashboards.html#url-option

erinreeves commented 4 years ago

semere:

  1. use one off script to upload via the api or something else
  2. based on ECK doc, you can upload by pointing to a url (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-dashboards.html)
  3. or may be you can mount the file to the kibana pod default path or even custom path to upload the file by enabling the configuration
  4. other approach is using filebeat (https://www.elastic.co/guide/en/beats/filebeat/current/directory-layout.html), which is what @erin.reeves wanted to try
erinreeves commented 4 years ago

Tried loading kibana dashboards via changing the configs in metricbeat as well as filebeat with no luck. Config setting tried: dashboards: enabled: true url: "https://github.com/cedardevs/onestop/blob/1297-load-kibana-dashboards/elastic-common/src/main/resources/dashboards/errors.zip"

Always get this error regardless of file format: 2020-07-23T20:18:43.227Z ERROR instance/beat.go:933 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing URL/file: Failed to unzip the archive: /tmp/tmp536853400/errors.zip: zip: not a valid zip file Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing URL/file: Failed to unzip the archive: /tmp/tmp536853400/errors.zip: zip: not a valid zip file

Tried several file formats. errors.zip (created via zip errors.zip errors.ndjson) errors.ndjson.zip (created via right clicking the file on Mac in file finder and selecting compress errors.ndjson) errors errors.jdjson (think was via gunzip, was first attempt and didn't work)

erinreeves commented 4 years ago

Found I was using the url to viewing the file in the pretty view in github, not the raw file. Changed the url to the raw file. Tried the: .ndjson - zip: not a valid zip file .zip - Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing URL/file: Too many directories under /tmp/tmp110104288 .ndjson.zip - same, too many directories.