sirmordred fails panels task on opendistro 1.11.0

marcofranssen commented 3 years ago

Elasticsearch and Kibana versions I'm using from opendistro which was suggested in https://github.com/chaoss/grimoirelab-sirmordred/issues/491

I'm running a docker-compose setup with following images.

amazon/opendistro-for-elasticsearch:1.11.0 amazon/opendistro-for-elasticsearch-kibana:1.11.0 bitergia/mordred:grimoirelab-0.2.50

$ utils/micro.py --panels --cfg /home/bitergia/conf/setup.cfg --debug
  [2020-11-05 12:25:45,036 - root - DEBUG] - Debug mode activated
  [2020-11-05 12:25:45,036 - sirmordred.config - DEBUG] - Reading conf files
  [2020-11-05 12:25:45,036 - sirmordred.config - DEBUG] - Reading conf files: /home/bitergia/conf/setup.cfg
  [2020-11-05 12:25:45,108 - urllib3.connectionpool - DEBUG] - Starting new HTTPS connection (1): elasticsearch:443
  [2020-11-05 12:25:45,149 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "GET / HTTP/1.1" 200 528
  [2020-11-05 12:25:45,152 - urllib3.connectionpool - DEBUG] - Starting new HTTPS connection (1): elasticsearch:443
  [2020-11-05 12:25:45,216 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "GET /.kibana HTTP/1.1" 200 4675
  [2020-11-05 12:25:45,219 - urllib3.connectionpool - DEBUG] - Starting new HTTP connection (1): kibana:5601
  [2020-11-05 12:25:46,518 - urllib3.connectionpool - DEBUG] - http://kibana:5601 "POST /api/kibana/settings/defaultIndex HTTP/1.1" 200 183
  [2020-11-05 12:25:47,651 - urllib3.connectionpool - DEBUG] - http://kibana:5601 "POST /api/kibana/settings/timepicker:timeDefaults HTTP/1.1" 200 183
  [2020-11-05 12:25:47,652 - sirmordred.task_panels - INFO] - Kibiter settings configured!
  [2020-11-05 12:25:47,653 - sirmordred.task_panels - INFO] - Dashboard panels, visualizations: uploading...
  [2020-11-05 12:25:47,653 - kidash.kidash - DEBUG] - Reading panels JSON file: /root/grimoirelab-sirmordred/src/grimoirelab-sigils/panels/json/overview.json
  [2020-11-05 12:25:47,653 - kidash.kidash - DEBUG] - Reading panel from directory: /root/grimoirelab-sirmordred/src/grimoirelab-sigils/panels/json/overview.json
  [2020-11-05 12:25:47,657 - kidash.kidash - DEBUG] - Panel detected.
  [2020-11-05 12:25:47,657 - kidash.kidash - DEBUG] - Retrieving dashboard Overview to check release date.
  [2020-11-05 12:25:47,657 - kidash.kidash - DEBUG] - Fetching dashboard Overview
  [2020-11-05 12:25:47,659 - urllib3.connectionpool - DEBUG] - Starting new HTTPS connection (1): elasticsearch:443
  [2020-11-05 12:25:47,682 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "GET /.kibana HTTP/1.1" 200 4675
  [2020-11-05 12:25:47,695 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "GET / HTTP/1.1" 200 528
  [2020-11-05 12:25:47,716 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "GET /.kibana/doc/dashboard:Overview HTTP/1.1" 404 77
  [2020-11-05 12:25:47,719 - kidash.kidash - DEBUG] - Can not find type dashboard item dashboard:Overview
  [2020-11-05 12:25:47,751 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "GET /.grimoirelab-sigils HTTP/1.1" 200 474
  [2020-11-05 12:25:47,951 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "GET /.grimoirelab-sigils/_search HTTP/1.1" 200 353
  [2020-11-05 12:25:47,953 - kidash.kidash - DEBUG] - Item dashboard Overview not found in Sigils index
  [2020-11-05 12:25:47,954 - kidash.kidash - DEBUG] - Release date for current item Overview is None.
  [2020-11-05 12:25:47,954 - kidash.kidash - DEBUG] - Release date for import item Overview is 2019-05-10T07:44:15.379531
  [2020-11-05 12:25:47,966 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "GET /.kibana HTTP/1.1" 200 4675
  [2020-11-05 12:25:47,970 - urllib3.connectionpool - DEBUG] - Starting new HTTPS connection (1): elasticsearch:443
  [2020-11-05 12:25:48,014 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "GET /.kibana HTTP/1.1" 200 4675
  [2020-11-05 12:25:48,017 - kidash.kidash - DEBUG] - Cleaning dashboard from studies vis
  [2020-11-05 12:25:48,019 - kidash.kidash - DEBUG] - Cleaning dashboard for dict_keys(['git', 'github', 'community', 'github-repos'])
  [2020-11-05 12:25:48,020 - kidash.kidash - DEBUG] - Cleaning dashboard from studies vis
  [2020-11-05 12:25:48,023 - kidash.kidash - DEBUG] - Not fixing height in Kibana >= 6 versions.
  [2020-11-05 12:25:48,023 - kidash.kidash - DEBUG] - Removing `release_date` from item dashboard:Overview since not allowed, and adding it to Sigils index
  [2020-11-05 12:25:48,678 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "POST /.grimoirelab-sigils/doc/dashboard:Overview HTTP/1.1" 200 185
  [2020-11-05 12:25:48,680 - kidash.kidash - DEBUG] - Release info added to Sigils index for dashboard:Overview
  [2020-11-05 12:25:48,728 - urllib3.connectionpool - DEBUG] - https://elasticsearch:443 "POST /.kibana/doc/dashboard:Overview HTTP/1.1" 400 279
Traceback (most recent call last):
  File "utils/micro.py", line 230, in <module>
    micro_mordred(args.cfg_path, args.backend_sections,
  File "utils/micro.py", line 72, in micro_mordred
    get_panels(config)
  File "utils/micro.py", line 138, in get_panels
    task.execute()
  File "/root/grimoirelab-sirmordred/sirmordred/task_panels.py", line 443, in execute
    self.create_dashboard(panel_file, data_sources=data_sources, strict=strict_loading)
  File "/root/grimoirelab-sirmordred/sirmordred/task_panels.py", line 424, in create_dashboard
    import_dashboard(es_enrich, kibana_url, panels_path, data_sources=data_sources, strict=strict)
  File "/root/grimoirelab-sirmordred/src/grimoirelab-kidash/kidash/kidash.py", line 1306, in import_dashboard
    feed_dashboard(json_to_import, elastic_url, kibana_url, es_index, data_sources, add_vis_studies)
  File "/root/grimoirelab-sirmordred/src/grimoirelab-kidash/kidash/kidash.py", line 1466, in feed_dashboard
    import_item_json(elastic, "dashboard", dashboard['dashboard']['id'],
  File "/root/grimoirelab-sirmordred/src/grimoirelab-kidash/kidash/kidash.py", line 717, in import_item_json
    res.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://admin:admin@elasticsearch/.kibana/doc/dashboard:Overview

Making a curl request shows the following

$ curl https://admin:admin@elasticsearch/.kibana/doc/dashboard:Overview -k
{"_index":".kibana_1","_type":"doc","_id":"dashboard:Overview","found":false}

Originally posted by @marcofranssen in https://github.com/chaoss/grimoirelab-sirmordred/issues/492#issuecomment-722347285

zhquan commented 3 years ago

I'm working on it

zhquan commented 3 years ago

Hi @marcofranssen,

Sorry for the late reply, kidash is available to use for ES 6.x but OpenDistro 1.11.0 uses ES 7.x due to that the structure of the index patterns and the panels have changed and we have to adapt kidash and the panels in sigils accordingly.

I'm still working on it and I'll keep you updated about it.

Best, Quan

marcofranssen commented 3 years ago

Thanks for the update, looking forward to the fix. Happy to test a prerelease of the mordred image as well.

On Thu, Nov 12, 2020 at 4:38 PM Quan Zhou notifications@github.com wrote:

Hi @marcofranssen https://github.com/marcofranssen,

Sorry for the late reply, kidash is available to use for ES 6.x but OpenDistro 1.11.0 uses ES 7.x due to that the structure of the index patterns and the panels have changed and we have to adapt kidash https://github.com/chaoss/grimoirelab-kidash and the panels in sigils https://github.com/chaoss/grimoirelab-sigils accordingly.

I'm still working on it and I'll keep you updated about it.

Best, Quan

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/grimoirelab-kidash/issues/40#issuecomment-726155789, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFJTTPUNAF3SH7OCTBCZQ3SPP6RDANCNFSM4TLJYDCQ .

marcofranssen commented 3 years ago

Hi @zhquan,

Just catching up with you on the progress of supporting OpenDistro. Would be great if you have a mordred docker image alpha release which I could test.

A long time ago I filed an issue on the kibiter project that told me to use opendistro instead.

Also found this blogpost. https://blog.bitergia.com/2020/06/08/kibiter-new-release-now-supporting-open-distro-for-elasticsearch/

What can I do to make it work?

zhquan commented 3 years ago

Hi @marcofranssen,

The SirMordred release is not ready yet, but you can do this:

On the setup.cfg disable panels (panels = false) in the section phases.
Use kidash to import the panels, index patterns stored in sigils but you must use these repos (you only have to do this the first time): 2.1. https://github.com/zhquan/grimoirelab-kidash/tree/opendistro 2.2. https://github.com/zhquan/grimoirelab-sigils/tree/opendistro/json

i.e for the dashboard git:

$ kidash -e https://admin:admin@localhost:9200 --import git-index-pattern.json
$ kidash -e https://admin:admin@localhost:9200 --import git.json

I will let you know when SirMordred is ready.

I hope it helps you.

Best, Quan

marcofranssen commented 3 years ago

@zhquan I have build following docker container to import dashboards

FROM python:3-alpine
RUN apk --no-cache add git
RUN git clone -b opendistro https://github.com/zhquan/grimoirelab-kidash.git /kidash
WORKDIR /kidash
RUN python3 setup.py install
RUN git clone -b opendistro https://github.com/zhquan/grimoirelab-sigils.git sigils
ENTRYPOINT [ "bin/kidash" ]

Now I run this inside the container

$ kidash -e http://admin:admin@elasticsearch --import sigils/json/git-index-pattern.json
2021-01-22 15:46:49,190 Index pattern git from ['git'] imported
$ kidash -e http://admin:admin@elasticsearch --import sigils/json/git.json
2021-01-22 15:46:53,752 Dashboard Git imported

But where in opendistro can I now find the menu-item with the dashboard? What am I missing?

zhquan commented 3 years ago

@marcofranssen - The menu is created by Bitergia on his own Kibana version named Kibiter. So if you want to use the menu you have to use Kibiter.

marcofranssen commented 3 years ago

@zhquan so how do I navigate to the dashboard in opendistro? I do see the index patterns, but struggle to find the dashboard itself.

zhquan commented 3 years ago

@marcofranssen - You will see the dashboard on the Dashboard section http://localhost:5601/app/dashboards#/list

marcofranssen commented 3 years ago

@zhquan Although kidash reports the index pattern and dashboard are created, I can't see them in Kibana. When I click Dashboards I'm redirected to the page to create a index pattern, so it seems the index pattern created from kidash isn't compatible with opendistro or isn't created accordingly.

Could you try with my setup?

docker-compose.yml

version: '3.7'

services:
  gateway:
    image: traefik:v2.3
    command:
      - --api.insecure=true
      - --entrypoints.http.address=:80
      - --entrypoints.https.address=:443
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --serversTransport.insecureSkipVerify=true
      - --log.level=DEBUG
    labels:
      traefik.enable: "true"
      traefik.http.routers.http-elasticsearch.rule: Host(`localhost`) && PathPrefix(`/es`) || Host(`elasticsearch`)
      traefik.http.routers.http-elasticsearch.entrypoints: http
      traefik.http.routers.http-elasticsearch.middlewares: redirect-to-https
      traefik.http.middlewares.redirect-to-https.redirectscheme.scheme: https
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  es01:
    image: amazon/opendistro-for-elasticsearch:${OPENDISTRO_VERSION}
    environment:
      cluster.name: grimoire-cluster
      node.name: es01
      discovery.seed_hosts: es01,es02
      cluster.initial_master_nodes: es01,es02
      bootstrap.memory_lock: "true"
      ES_JAVA_OPTS: -Xms${ES_HEAP_SIZE}m -Xmx${ES_HEAP_SIZE}m
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    volumes:
      - "./es-data/es01:/usr/share/elasticsearch/data:delegated"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    labels:
      traefik.enable: "true"
      traefik.http.routers.elasticsearch.entrypoints: https
      traefik.http.routers.elasticsearch.rule: (Host(`localhost`) || Host(`${TRAEFIK_HOST}`)) && PathPrefix(`/es`) || Host(`elasticsearch`)
      traefik.http.routers.elasticsearch.middlewares: es-stripprefix
      traefik.http.routers.elasticsearch.tls:
      traefik.http.middlewares.es-stripprefix.stripprefix.prefixes: /es
      traefik.http.services.elasticsearch.loadbalancer.server.port: 9200
      traefik.http.services.elasticsearch.loadbalancer.server.scheme: https

  es02:
    image: amazon/opendistro-for-elasticsearch:${OPENDISTRO_VERSION}
    environment:
      cluster.name: grimoire-cluster
      node.name: es02
      discovery.seed_hosts: es01,es02
      cluster.initial_master_nodes: es01,es02
      bootstrap.memory_lock: "true"
      ES_JAVA_OPTS: -Xms${ES_HEAP_SIZE}m -Xmx${ES_HEAP_SIZE}m
    volumes:
      - "./es-data/es02:/usr/share/elasticsearch/data:delegated"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    labels:
      traefik.enable: "true"
      traefik.http.routers.elasticsearch.entrypoints: https
      traefik.http.routers.elasticsearch.rule: (Host(`localhost`) || Host(`${TRAEFIK_HOST}`)) && PathPrefix(`/es`) || Host(`elasticsearch`)
      traefik.http.routers.elasticsearch.middlewares: es-stripprefix
      traefik.http.routers.elasticsearch.tls:
      traefik.http.middlewares.es-stripprefix.stripprefix.prefixes: /es
      traefik.http.services.elasticsearch.loadbalancer.server.port: 9200
      traefik.http.services.elasticsearch.loadbalancer.server.scheme: https

  mariadb:
    image: "mariadb:${MARIADB_VERSION}"
    restart: always
    env_file:
      - .env
    command: --wait_timeout=2592000 --interactive_timeout=2592000 --max_connections=300
    ports:
      - "3306:3306"
    volumes:
      - "./mariadb-data:/var/lib/mysql:delegated"

  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:${OPENDISTRO_VERSION}
    environment:
      ELASTICSEARCH_URL: https://elasticsearch
      ELASTICSEARCH_HOSTS: https://elasticsearch
    links:
      - "gateway:elasticsearch"
    depends_on:
      - gateway
    labels:
      traefik.enable: "true"
      traefik.http.routers.dashboard.entrypoints: http
      traefik.http.routers.dashboard.rule: Host(`localhost`) || Host(`${TRAEFIK_HOST}`)
      traefik.http.services.dashboard.loadbalancer.server.port: 5601

  kidash:
    image: kidash
    build: opendistro-dashboards
    entrypoint: /usr/bin/env sh
    command: |-
      -c '
        kidash --debug -e http://admin:admin@elasticsearch --import sigils/json/git-index-pattern.json
        kidash --debug -e http://admin:admin@elasticsearch --import sigils/json/git.json
      '
    depends_on:
      - kibana
      - gateway
    links:
      - "gateway:elasticsearch"

  hatstall:
    image: grimoirelab/hatstall:latest
    environment:
      - DATABASE_DIR=/db/
      - ADMIN_USER=admin
      - ADMIN_PASS=admin
    volumes:
      - ./conf/grimoirelab/apache-hatstall.conf:/home/grimoirelab/apache-hatstall.conf
      - ./conf/grimoirelab/shdb.cfg:/home/grimoirelab/shdb.cfg
    links:
      - mariadb
    depends_on:
      - gateway
    labels:
      traefik.enable: "true"
      traefik.http.routers.hatstall.entrypoints: http
      traefik.http.routers.hatstall.rule: (Host(`localhost`) || Host(`${TRAEFIK_HOST}`)) && (PathPrefix(`/identities/hatstall`) || PathPrefix(`/static/hatstall`) || PathPrefix(`/accounts/login`))
      traefik.http.services.hatstall.loadbalancer.server.port: 80

  mordred:
    restart: on-failure:5
    image: bitergia/mordred:grimoirelab-${GRIMOIRELAB_VERSION}
    volumes:
      - ./conf/grimoirelab/setup.cfg:/home/bitergia/conf/setup.cfg
      - ./conf/grimoirelab/credentials.cfg:/home/bitergia/conf/credentials.cfg
      - ./conf/grimoirelab/aliases.json:/home/bitergia/conf/aliases.json
      - ./conf/grimoirelab/projects.json:/home/bitergia/conf/projects.json
      - ./conf/grimoirelab/orgs.json:/home/bitergia/conf/organizations.json
      - ./conf/grimoirelab/identities.yml:/home/bitergia/conf/identities.yml
      - ./logs:/home/bitergia/logs:delegated
      - ./.perceval:/home/bitergia/.perceval:delegated
      - ./mordred-stage.sh:/home/bitergia/stage
    links:
      - "gateway:elasticsearch"
    depends_on:
      - mariadb
      - gateway

opendistro-dashboards/Dockerfile

FROM python:3-alpine
RUN apk --no-cache add git
RUN git clone -b opendistro https://github.com/zhquan/grimoirelab-kidash.git /kidash
WORKDIR /kidash
RUN python3 setup.py install
RUN git clone -b opendistro https://github.com/zhquan/grimoirelab-sigils.git sigils
ENTRYPOINT [ "bin/kidash" ]

Howto run

Put in place the grimorelab configs as defined in the volumes in the docker-compose setup. Left that out here as you probably know better what to put there.

docker-compose up -d
# wait for kibana to be up at http://localhost
docker-compose up kidash

Elasticsearch nodes can be accessed at http://localhost/es (loadbalanced via traefik)
Kidash uses the internal alias http://elasticsearch

Also see here the debug log from kidash

kidash_1    | 2021-01-26 10:51:00,394 Debug mode activated
kidash_1    | 2021-01-26 10:51:00,394 Reading panels JSON file: sigils/json/git-index-pattern.json
kidash_1    | 2021-01-26 10:51:00,394 Reading panel from directory: sigils/json/git-index-pattern.json
kidash_1    | 2021-01-26 10:51:00,394 Index-Pattern detected.
kidash_1    | 2021-01-26 10:51:00,451 Removing `release_date` from item index-pattern:git since not allowed, and adding it to Sigils index
kidash_1    | 2021-01-26 10:51:05,527 Release info added to Sigils index for index-pattern:git
kidash_1    | 2021-01-26 10:51:05,594 Reading panel from directory: sigils/json/git-index-pattern.json
kidash_1    | 2021-01-26 10:51:05,594 Index pattern git from ['git'] imported
kidash_1    | 2021-01-26 10:51:05,921 Debug mode activated
kidash_1    | 2021-01-26 10:51:05,921 Reading panels JSON file: sigils/json/git.json
kidash_1    | 2021-01-26 10:51:05,921 Reading panel from directory: sigils/json/git.json
kidash_1    | 2021-01-26 10:51:05,922 Panel detected.
kidash_1    | 2021-01-26 10:51:05,986 Cleaning dashboard from studies vis
kidash_1    | 2021-01-26 10:51:05,987 Not fixing height in Kibana >= 6 versions.
kidash_1    | 2021-01-26 10:51:05,987 Removing `release_date` from item dashboard:Git since not allowed, and adding it to Sigils index
kidash_1    | 2021-01-26 10:51:10,586 Release info added to Sigils index for dashboard:Git
kidash_1    | 2021-01-26 10:51:10,830 Reading panel from directory: sigils/json/git.json
kidash_1    | 2021-01-26 10:51:10,831 Dashboard Git imported

zhquan commented 3 years ago

@marcofranssen - you have to tell kidash your .kibana_xxx_admin_x index:

kidash -e http://admin:admin@elasticsearch --import sigils/json/git-index-pattern.json --kibana .kibana_xxx_admin_x

marcofranssen commented 3 years ago

@zhquan thanks that did the trick. Looking forward to have this natively in mordred.

vchrombie commented 2 years ago

@zhquan @marcofranssen thanks for participating in the discussion.

Do you think we can close this issue?

chaoss / grimoirelab-kidash

sirmordred fails panels task on opendistro 1.11.0 #40