docker run error and elasticsearch 7.7.0

papioyue commented 4 years ago

15:13:58.664Z ERROR elastalert-server: ProcessController: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main

15:13:58.665Z ERROR elastalert-server: ProcessController: "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/opt/elastalert/elastalert/elastalert.py", line 1929, in sys.exit(main(sys.argv[1:])) File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main

15:13:58.665Z ERROR elastalert-server: ProcessController: client.start() File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start

15:13:58.665Z ERROR elastalert-server: ProcessController: self.run_all_rules() File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules

15:13:58.666Z ERROR elastalert-server: ProcessController: self.send_pending_alerts() File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts

15:13:58.666Z ERROR elastalert-server: ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)

15:13:58.666Z ERROR elastalert-server: ProcessController: File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts

15:13:58.667Z ERROR elastalert-server: ProcessController: size=1000)

15:13:58.667Z ERROR elastalert-server: ProcessController: File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped return func(*args, params=params, **kwargs) TypeError: search() got an unexpected keyword argument 'doc_type'

15:13:58.711Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1 15:13:58.711Z INFO elastalert-server: Server: Stopping server 15:13:58.711Z INFO elastalert-server: ProcessController: ElastAlert is not running 15:13:58.711Z INFO elastalert-server: Server: Server stopped. Bye!

nsano-rururu commented 4 years ago

ElastAlert Server Docker Images

Docker image name	tag	ElastAlert	Elasticsearch 7 Support	Remarks
bitsensor/elastalert	2.0.1	0.1.39	×
bitsensor/elastalert	lastet	0.1.39	×
bitsensor/elastalert	3.0.0-beta.0	0.2.0b2	〇
bitsensor/elastalert	3.0.0-beta.1	0.2.0b2	〇
servercentral/elastalert	latest	0.2.1	〇	bitsensor/elastalert fork Customize bugfix Python 3.6
daichi703n/elastalert	0.2.1-dev2	0.2.1	〇	servercentral/elastalert fork Customize bugfix Python 3.6
praecoapp/elastalert-server	latest	0.2.4	〇	servercentral/elastalert fork Customize bugfix Library Update Python 3.8

example

elasticsearch:7.7.0 kibana:7.7.0 praecoapp/praeco:latest praecoapp/elastalert-server:latest

praeco/nginx_config/default.conf https://github.com/johnsusek/praeco/blob/master/nginx_config/default.conf praeco/nginx_config/nginx.conf https://github.com/johnsusek/praeco/blob/master/nginx_config/nginx.conf praeco/public/favicon.ico https://github.com/johnsusek/praeco/blob/master/public/favicon.ico praeco/public/index.html https://github.com/johnsusek/praeco/blob/master/public/index.html praeco/public/js/cron-ui.min.js https://github.com/johnsusek/praeco/blob/master/public/js/cron-ui.min.js

/home/sano/dkwork2/es
|--Dockerfiles
|  |--Dockerfile.elastalert
|--docker-compose.yml
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|--kibana
|  |--config
|  |  |--kibana.yml
|--praeco
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--api.config.json
|  |  |--elastalert.yaml
|  |--nginx_config
|  |  |--default.conf
|  |  |--nginx.conf
|  |--public
|  |  |--favicon.ico
|  |  |--index.html
|  |  |--js
|  |  |  |--cron-ui.min.js
|  |  |--praeco.config.json
|  |--rule_templates
|  |--rules

Dockerfiles/Dockerfile.elastalert

FROM praecoapp/elastalert-server:latest

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD praeco/bin/elastalert-start.sh /usr/local/bin/
ADD praeco/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx512m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.7.0
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile.elastalert
    image: elastalert-server:3.0.0
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./praeco/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./praeco/config/api.config.json:/opt/elastalert-server/config/config.json
      - ./praeco/rules:/opt/elastalert/rules
      - ./praeco/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  praeco:
    container_name: praeco
    image: praecoapp/praeco:latest
    ports:
      - 8080:8080
    depends_on:
      - elastalert
    restart: always
    volumes:
      - ./praeco/public/praeco.config.json:/var/www/html/praeco.config.json
      - ./praeco/nginx_config/nginx.conf:/etc/nginx/nginx.conf
      - ./praeco/nginx_config/default.conf:/etc/nginx/conf.d/default.conf
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:8080 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

praeco/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

praeco/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

praeco/config/api.config.json

Default settings

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "es_username": "",
  "es_password": "",
  "es_ssl": false,
  "writeback_index": "praeco_elastalert_status"
}

praeco/config/elastalert.yaml

Default settings

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 60

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: praeco_elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

skip_invalid: True

profile: default

praeco/public/praeco.config.json

Default settings

{
  "appUrl": "http://127.0.0.1:8080",
  "errorLoggerUrl": "",
  "hidePreconfiguredFields": []
}

setting

cd /home/sano/dkwork2/es
chmod 777 es/data
chmod -R 777 praeco/rules praeco/rule_templates
docker-compose up -d

nnovaes commented 4 years ago

@nsano-rururu if it works in 7.7, is there any particular reason why 7.0 is still listed as requirement on setup?

https://github.com/Yelp/elastalert/blob/1dc4f30f30d39a689f419ce19c7e2e4d67a50be3/requirements.txt#L8

nnovaes commented 4 years ago

thanks! i'm looking at fixing CVEs opened in my docker image and 7.0.0 has this CVE which is only closed on versions >7.2.1. i'll make some tests to see if i can upgrade it.

papioyue commented 3 years ago

yes发自我的华为手机-------- 原始邮件 --------发件人： Naoyuki Sano notifications@github.com日期： 2020年12月16日周三清晨6:28收件人： Yelp/elastalert elastalert@noreply.github.com抄送： papioyue 16619765933@163.com, Mention mention@noreply.github.com主题： Re: [Yelp/elastalert] docker run error and elasticsearch 7.7.0 (#2995) Please close if resolved

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

oldthreefeng commented 3 years ago

how can i use this by es 6.x.x ?

Yelp / elastalert

docker run error and elasticsearch 7.7.0 #2995