Plugin installation was unsuccessful due to error "Plugin elastalert-kibana-plugin [7.5.0] is incompatible with Kibana [7.5.2]"

Yunamii commented 4 years ago

Hello,

I have kibana 7.5.2 and I whould install elastalert on Kibana. When I install the latest version of elastalert kibana plugin, I have the error message below:

./kibana-plugin install https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip --allow-root Attempting to transfer from https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip Transferring 25617568 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation was unsuccessful due to error "Plugin elastalert-kibana-plugin [7.5.0] is incompatible with Kibana [7.5.2]"

Is there a future deployement for the 7.5.2 version?

whataboutpereira commented 4 years ago

Unfortunately the same issue with any newer version (like the current 7.6.1).

nsano-rururu commented 4 years ago

@Yunamii

Currently, zip files for Kibana 7.5.1 and Kibana 7.5.2 are not released, so you need to download the zip for Kibana 7.5.0 and edit package.json.

How to create and install a zip file for kiban 7.5.2 is as follows

cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.5.2.zip
unzip elastalert-kibana-plugin-1.1.0-7.5.2.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.5\.2/g" kibana/elastalert-kibana-plugin/package.json
zip elastalert-kibana-plugin-1.1.0-7.5.2.zip kibana/elastalert-kibana-plugin/package.json
rm -rf kibana
cd /usr/share/kibana/bin
./kibana-plugin install --allow-root file:///tmp/elastalert-kibana-plugin-1.1.0-7.5.2.zip

shijeshk commented 4 years ago

Still running into the same problem

Attempting to transfer from file:///tmp/new/elastalert-kibana-plugin-1.1.0-7.5.2.zip Transferring 25617568 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation was unsuccessful due to error "Plugin elastalert-kibana-plugin [7.5.0] is incompatible with Kibana [7.5.2]"

nsano-rururu commented 4 years ago

@shijeshk

I put the elastalert-kibana-plugin-1.1.0-7.5.2.zip file in my github repository, can you check if the problem occurs again? https://github.com/nsano-rururu/docker-sample/blob/master/elastalert-kibana-plugin/kibana/plugin/elastalert-kibana-plugin-1.1.0-7.5.2.zip

shijeshk commented 4 years ago

Thanks @nsano-rururu I am able to install now.

nsano-rururu commented 4 years ago

@shijeshk

Thank you for confirming the install.

nsano-rururu commented 4 years ago

@Yunamii

Is the problem solved? . If there are no additional questions, please close.

markgaolei commented 4 years ago

@nsano-rururu Could i use above command to build 7.6.0 version?

nsano-rururu commented 4 years ago

@nsano-rururu Could i use above command to build 7.6.0 version?

@markgaolei

cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js
mv elastalert.js elastalert-server-routes.js
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.6.0.zip
unzip elastalert-kibana-plugin-1.1.0-7.6.0.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.6\.0/g" kibana/elastalert-kibana-plugin/package.json
mkdir -p kibana/elastalert-kibana-plugin/server/routes/
cp /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js
zip elastalert-kibana-plugin-1.1.0-7.6.0.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js
rm -rf kibana
rm elastalert-server-routes.js
cd /usr/share/kibana/bin
./kibana-plugin install --allow-root file:///tmp/elastalert-kibana-plugin-1.1.0-7.6.0.zip

mismailzz commented 4 years ago

@nsano-rururu After successful installation of Elastalert for kibana 7.6.2, I got an error on kibana. Error: Kibana server is not ready yet

nsano-rururu commented 4 years ago

@mismailzz

・Is Elasticsearch also known as 7.6.2?

・Is Elasticsearch started?

・Is setting added to kibana.yml?

# defaults to localhost
elastalert-kibana-plugin.serverHost: ip address or hostname
# defaults to 3030
elastalert-kibana-plugin.serverPort: 3030
# defaults to true
#elastalert-kibana-plugin.enabled
# defaults to false 
#elastalert-kibana-plugin.serverSsl

・Did the elastalert kibana plugin execute the following commands for Kibana 7.6.2? . Did you execute something different from the following command?

cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js
mv elastalert.js elastalert-server-routes.js
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.6.2.zip
unzip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.6\.2/g" kibana/elastalert-kibana-plugin/package.json
mkdir -p kibana/elastalert-kibana-plugin/server/routes/
cp /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js
zip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js
rm -rf kibana
rm elastalert-server-routes.js
cd /usr/share/kibana/bin
./kibana-plugin install --allow-root file:///tmp/elastalert-kibana-plugin-1.1.0-7.6.2.zip

whataboutpereira commented 4 years ago

After successful installation of Elastalert for kibana 7.6.2, I got an error on kibana. Error: Kibana server is not ready yet

That usually implies Kibana is still starting up.

nsano-rururu commented 4 years ago

If the message "Kibana server is not ready yet" is displayed, it is possible that there is a problem with the config or other items that are still starting up, or the startup is interrupted.

I've often run into an issue where I forget to add the settings to kibana.yml and it doesn't start. . .

After that, even if you are in trouble if you do not work with machine specifications (CPU, memory) and JVM memory settings set too little. . .

mismailzz commented 4 years ago

@nsano-rururu, Yes I'm using Elasticsearch version 7.6.2, it is running perfectly and during the installation of Elastalert I changed to version 7.6.0 to 7.6.2. The only thing that I skipped was the Elastic plugin information in the Kibana.yml file. Now when I added this configuration in Kibana.yml file then I got the same issue. I added the configuration like this at the end of file

# defaults to localhost
elastalert-kibana-plugin.serverHost: "localhost"
# defaults to 3030
elastalert-kibana-plugin.serverPort: 5601
# defaults to true
#elastalert-kibana-plugin.enabled
# defaults to false 
#elastalert-kibana-plugin.serverSsl

One last thing is that after some time while kibana is running the machine get stuck.

nsano-rururu commented 4 years ago

@mismailzz

As a result of trying with Docker, the plugin of ElastAlert worked with Kibana 7.6.2

Elasticsearch 7.6.2 Kibana 7.6.2 kibana-elastalert-plugin bitsensor/elastalert:3.0.0-beta.0

AWS Workspace Amazon Linux2 2 vCPU、7.5GiB

/home/user/docker-wk2
|--docker-compose.yml
|--Dockerfiles
|  |--Dockerfile-elastalert
|
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|
|--kibana
|  |--config
|  |  |--kibana.yml
|  |--plugin
|  |  |--kibana-elastealert-plugin-1.1.0-7.6.2.zip
|
|--elastalert
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--config.json
|  |  |--elastalert-test.yaml
|  |  |--elastalert.yaml
|  |--rule_templates
|  |--rules

cd 
mkdir docker-wk2
cd docker-wk2
touch docker-compose.yml
mkdir Dockerfiles
touch Dockerfiles/Dockerfile-elastalert
touch Dockerfiles/Dockerfile-kibana
mkdir -p es/config
mkdir -p es/data
touch es/config/elasticsearch.yml
mkdir -p kibana/config
mkdir -p kibana/plugin
touch kibana/config/kibana.yml
mkdir -p elastalert/bin
mkdir -p elastalert/config
mkdir -p elastalert/rule_templates
mkdir -p elastalert/rules
touch elastalert/bin/elastalert-start.sh
touch elastalert/bin/elastic_search_status.sh
touch elastalert/config/config.json
touch elastalert/config/elastalert-test.yaml
touch elastalert/config/elastalert.yaml

# Download necessary files
cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js

# update elasticsearch package to 7.6.2
mv elastalert.js elastalert-server-routes.js
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.6.2.zip
unzip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.6\.2/g" kibana/elastalert-kibana-plugin/package.json
mkdir -p kibana/elastalert-kibana-plugin/server/routes/
mv /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js
zip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js

# delete remaining directory
rm -rf kibana

# copy the created package to your workspace
mv /tmp/elastalert-kibana-plugin-1.1.0-7.6.2.zip /home/user/docker-wk2/kibana/plugin

cd
cd docker-wk2

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx256m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.6.2
    command: sh -c './bin/kibana-plugin list | grep elastalert-kibana-plugin@1.1.0; result=`echo $$?`; if [ $$result = 1 ]; then  ./bin/kibana-plugin install file:///usr/share/kibana/work/elastalert-kibana-plugin-1.1.0-7.6.2.zip && exec /usr/local/bin/kibana-docker; else exec /usr/local/bin/kibana-docker; fi'
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
      - ./kibana/plugin:/usr/share/kibana/work
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile-elastalert
    image: elastalert:3.0.0-beta.0
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
      - kibana
    restart: always
    volumes:
      - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./elastalert/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml
      - ./elastalert/config/config.json:/opt/elastalert-server/config/config.json
      - ./elastalert/rules:/opt/elastalert/rules
      - ./elastalert/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

# elastalert-kibana-plugin
elastalert-kibana-plugin.serverHost: elastalert
elastalert-kibana-plugin.serverPort: 3030

Dockerfiles/Dockerfile-elastalert

FROM bitsensor/elastalert:3.0.0-beta.0

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

elastalert/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

elastalert/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

elastalert/config/config.json

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "writeback_index": "elastalert_status"
}

elastalert/config/elastalert-test.yml

# NOTE: This config is used when testing a rule

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

elastalert/config/elastalert.yml

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

$ chmod 777 es/data
$ chmod 777 elastalert/rules
$ chmod 777 elastalert/rule_templates

$ docker-compose up -d

$ docker ps

CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                   PORTS                                            NAMES
b2f0a31c5b36        elastalert:3.0.0-beta.0                               "/usr/local/bin/elas…"   6 minutes ago       Up 6 minutes (healthy)   0.0.0.0:3030->3030/tcp, 0.0.0.0:3333->3333/tcp   elastalert
321dc2d22b00        docker.elastic.co/kibana/kibana:7.6.2                 "/usr/local/bin/dumb…"   6 minutes ago       Up 6 minutes (healthy)   0.0.0.0:5601->5601/tcp                           kibana
2ecd8dc1934a        docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/local/bin/dock…"   6 minutes ago       Up 6 minutes (healthy)   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch

mismailzz commented 4 years ago

@nsano-rururu buddy, Thank you so much. I will try through docker and let you know, soon. Allah bless you!

nsano-rururu commented 4 years ago

@mismailzz

Reference information. You may already know. .

Elasticsearch will run out of Java heap memory if you continue to input data with the index open. circuit_breaking_exception may occur → Elasticsearch heap size change → Elasticsearch indices.breaker.total.limit changed → Close unnecessary index data Or Delete unnecessary Index data ILM: Manage the index lifecycle

ElastAlert also falls out of memory. Probably, it is possible that the open data of the index to be searched once is expanded in the memory and searched.

nsano-rururu commented 4 years ago

@mismailzz

I could also make a Docker image with Kibana's elastalert-kibana-plugin installed in advance, so I will share the information.

・docker image create (install elastalert-kibana-plugin on Kibana)

cd
mkdir -p docker-image-build/kibana
cd docker-image-build/kibana
mkdir -p kibana/elastalert-kibana-plugin/server/routes
cd kibana/elastalert-kibana-plugin/server/routes
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js
cd 
cd docker-image-build/kibana
mkdir -p elastalert-kibana-plugin
cd elastalert-kibana-plugin
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
cd
cd docker-image-build/kibana
touch Dockerfile-elastalert-kibana-plugin-1.1.0-7.6.2

・Dockerfile-elastalert-kibana-plugin-1.1.0-7.6.2

FROM docker.elastic.co/kibana/kibana:7.6.2

USER root

RUN yum install -y \
    zip \
    unzip && \
    rm -rf /var/cache/yum/* && \
    yum clean all

ADD elastalert-kibana-plugin/elastalert-kibana-plugin-1.1.0-7.5.0.zip /tmp/elastalert-kibana-plugin.zip
ADD kibana/elastalert-kibana-plugin/server/routes/elastalert.js /tmp/elastalert-server-routes.js

RUN cd /tmp && \
      unzip elastalert-kibana-plugin.zip kibana/elastalert-kibana-plugin/package.json && \
      sed -i "s/7\.5\.0/7\.6\.2/g" kibana/elastalert-kibana-plugin/package.json && \
      mkdir -p kibana/elastalert-kibana-plugin/server/routes/ && \
      cp /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js && \
      zip elastalert-kibana-plugin.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js && \
      cd /usr/share/kibana/plugins && \
      /usr/share/kibana/bin/kibana-plugin install file:///tmp/elastalert-kibana-plugin.zip --allow-root && \
      rm -rf /tmp/elastalert-kibana-plugin.zip /tmp/elastalert.js /tmp/kibana

USER kibana

docker build -t kibana:7.6.2 -f Dockerfile-elastalert-kibana-plugin-1.1.0-7.6.2 .

・docker-compose.yml update

add「image: kibana:7.6.2」 coment out「image: docker.elastic.co/kibana/kibana:7.6.2」 and 「command:・・・」

kibana:
    container_name: kibana
    image: kibana:7.6.2
    # image: docker.elastic.co/kibana/kibana:7.6.2
    # command: sh -c './bin/kibana-plugin list | grep elastalert-kibana-plugin@1.1.0; result=`echo $$?`; if [ $$result = 1 ]; then  ./bin/kibana-plugin install file:///usr/share/kibana/work/elastalert-kibana-plugin-1.1.0-7.6.2.zip && exec /usr/local/bin/kibana-docker; else exec /usr/local/bin/kibana-docker; fi'

mismailzz commented 4 years ago

@nsano-rururu Issue resolved, Thank You So Much!

bitsensor / elastalert-kibana-plugin

Plugin installation was unsuccessful due to error "Plugin elastalert-kibana-plugin [7.5.0] is incompatible with Kibana [7.5.2]" #146