search

Yelp / elastalert

Easy & Flexible Alerting With ElasticSearch
https://elastalert.readthedocs.org
Apache License 2.0
8k stars 1.73k forks source link

Match_body is not parsing #3093

Closed aniketpant1 closed 4 months ago

aniketpant1 commented 3 years ago

When i ran elastalert_create_index it show me this

Elastic Version: 7.10.1 Reading Elastic 6 index mappings: Reading index mapping 'es_mappings/6/silence.json' Reading index mapping 'es_mappings/6/elastalert_status.json' Reading index mapping 'es_mappings/6/elastalert.json' Reading index mapping 'es_mappings/6/past_elastalert.json' Reading index mapping 'es_mappings/6/elastalert_error.json' Traceback (most recent call last): File "/usr/local/bin/elastalert-create-index", line 11, in load_entry_point('elastalert==0.2.4', 'console_scripts', 'elastalert-create-index')() File "/usr/local/lib/python3.6/site-packages/elastalert-0.2.4-py3.6.egg/elastalert/create_index.py", line 264, in main create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index) File "/usr/local/lib/python3.6/site-packages/elastalert-0.2.4-py3.6.egg/elastalert/create_index.py", line 72, in create_index_mappings body=es_index_mappings['past_elastalert'], include_type_name=True) File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/client/utils.py", line 84, in _wrapped return func(*args, params=params, **kwargs) File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/client/indices.py", line 321, in put_mapping "PUT", _make_path(index, "_mapping", doc_type), params=params, body=body File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/transport.py", line 318, in perform_request status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout) File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/connection/http_requests.py", line 91, in perform_request self._raise_error(response.status_code, raw_data) File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/connection/base.py", line 131, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) elasticsearch.exceptions.TransportError: TransportError(500, 'mapper_exception', "the [enabled] parameter can't be updated for the object mapping [match_body]")

and this is my template PUT /_template/elastalert { "index_patterns": ["elastalert*"], "settings": { "index": { "mapping": { "total_fields.limit": "10000" } }, "number_of_replicas": 0, "number_of_shards": 3 } , "mappings": { "properties": { "match_body": { "properties": { "z_logstash_pipeline": { "type": "keyword" }, "etl_pipeline": { "type": "keyword" } } } } } }

nsano-rururu commented 3 years ago

I checked with the following version with Docker, but the problem did not reproduce. There is no problem with alert notification.

Docker ・Elasticsearch 7.10.1 ・Kibana 7.10.1 ・Elast ・praecoapp/elastalert-server:latest 　- ElastAlert 0.2.4 + Bugfix etc ・praecoapp/praeco:latest

Giving Elasticsearch at  time to start...
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!

> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

10:13:31.280Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
10:13:31.281Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
10:13:31.290Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
10:13:31.306Z  INFO elastalert-server: Router:  Listening for GET request on /.
10:13:31.306Z  INFO elastalert-server: Router:  Listening for GET request on /status.
10:13:31.307Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
10:13:31.307Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
10:13:31.312Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
10:13:31.312Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
10:13:31.313Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
10:13:31.313Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
10:13:31.313Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
10:13:31.313Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
10:13:31.317Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
10:13:31.317Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
10:13:31.317Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
10:13:31.317Z  INFO elastalert-server: Router:  Listening for POST request on /test.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for GET request on /config.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for POST request on /config.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for POST request on /download.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
10:13:31.319Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
10:13:31.319Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
10:13:31.319Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
10:13:31.319Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
10:13:31.320Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
10:13:31.320Z  INFO elastalert-server: Router:  Listening for GET request on /config.
10:13:31.331Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
10:13:31.331Z  INFO elastalert-server: ProcessController:  Creating index
10:13:35.335Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.10.1
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index praeco_elastalert_status created
    Done!

10:13:35.335Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
10:13:35.336Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
10:13:35.347Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 43)
10:13:35.350Z  INFO elastalert-server: Server:  Server listening on port 3030
10:13:35.390Z  INFO elastalert-server: Server:  Websocket listening on port 3333
10:13:35.394Z  INFO elastalert-server: Server:  Server started
10:13:57.531Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:14:28.007Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:14:58.405Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:15:28.732Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:15:38.131Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:15:38.145Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
10:15:38.150Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
10:15:38.170Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules'.
10:15:38.172Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/templates'.
10:15:59.042Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:16:23.849Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
10:16:29.358Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:16:45.802Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
10:16:55.418Z  INFO elastalert-server: Routes:  Successfully handled POST request for '/rules/:id'.
10:16:55.499Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
10:16:55.502Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
10:16:55.525Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules'.
10:16:55.528Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/templates'.
10:16:59.705Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:17:30.059Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.

キャプチャ

aniketpant1 commented 3 years ago

I am using elastalert and sigma for transforming windows event logs to elastalert signature.I am using HELK but the component i am using is ELK+kafka+elastalert+sigma.The rules folder contain helk This is working fine

This is my script which i've copied but i made some changes.This script is for transforming sigma rules to elastalert signature pull_sigma.sh

!/bin/bash

helk_sigmac=/usr/share/sigma/sigmac/sigmac-config.yml ESALERT_HOME=/usr/share/elastalert/

* Unsupported SIGMA Functions *****

Unsupported feature "near" aggregation operator not yet implemented https://github.com/Neo23x0/sigma/issues/209

SIGMAremoveNearRules() { if grep --quiet -E "\s+condition/\s+.*\s+|\s+near\s+" "$1"; then echo -e "Skipping incompatible rule $1, reference: https://github.com/Neo23x0/sigma/issues/209"

rm "$1"

    return 0
else
  return 1
fi

}

Transforming every Windows SIGMA rule to elastalert rules

echo " " echo "Translating SIGMA rules to Elastalert format.." echo "------------------------------------------------" echo " " rule_counter=0

Windows rules

for rule_category in rules/windows/ ; do echo " " echo -e "${HELK_INFO_TAG} Working on Folder: $rule_category:" echo "-------------------------------------------------------------" if [[ "$rule_category" == "rules/windows/process_creation" ]]; then for rule in "${rule_category}"/ ; do if [[ ${rule} != "rules/windows/process_creation/win_mal_adwind.yml" ]]; then if SIGMAremoveNearRules "$rule"; then continue else echo "[+++] Processing Windows process creation rule: $rule .." tools/sigmac -t elastalert -c tools/config/generic/sysmon.yml -c "${helk_sigmac}" --backend-option timestamp_field=etl_processed_time -o "${ESALERT_HOME}"/rules/sigmasysmon"$(basename "${rule}")" "$rule"

Give unique rule name for sysmon

                sed -i 's/^name: /name: Sysmon_/' "${ESALERT_HOME}"/rules/sigma_sysmon_"$(basename "${rule}")"
                tools/sigmac -t elastalert -c tools/config/generic/windows-audit.yml -c "${helk_sigmac}" --backend-option timestamp_field=etl_processed_time -o "${ESALERT_HOME}"/rules/sigma_"$(basename "${rule}")" "$rule"
                rule_counter=$[$rule_counter +1]
            fi
        fi
    done
else
    for rule in "${rule_category}"/* ; do
        if SIGMAremoveNearRules "$rule"; then
            continue
        else
            echo "[+++] Processing additional Windows rule: $rule .."
            tools/sigmac -t elastalert -c "${helk_sigmac}" --backend-option timestamp_field=etl_processed_time -o "${ESALERT_HOME}"/rules/sigma_"$(basename "${rule}")" "$rule"
            rule_counter=$[$rule_counter +1]
        fi
    done
fi

done

But problems comes when i have sigma This is all 800(approx) rules causes elasticsearch lead to crash Thats why i move all the 800 rules to another directory and at the end i have few rules(that above picture which start with helk_) to run which does not lead to elasticsearch to crash by crash means elasticsearch behave abnormal it consumes full heap size and when i don't run those 800 rules it works perfectly I am not using docker.

nsano-rururu commented 3 years ago

Why don't you delete the index of elastalert created in Elasticsearch with elastalert-create-index and try to execute it again?

nsano-rururu commented 3 years ago

Is python 3.6?

aniketpant1 commented 3 years ago

yes it is python 3.6 it is creating index.

nsano-rururu commented 3 years ago

Is it possible to check what happens when the following pull request changes are reflected? https://github.com/Yelp/elastalert/pull/3016

aniketpant1 commented 3 years ago

What pull request ? in my log file of elastalert it show me Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/connection/http_requests.py", line 77, in perform_request response = self.session.send(prepared_request, send_kwargs) File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send r = adapter.send(request, kwargs) File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 529, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPConnectionPool(host='elastic', port=9200): Read timed out. (read timeout=20) ERROR:root:Error running query: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='elastic', port=9200): Read timed out. (read timeout=20)) WARNING:elasticsearch:GET http://elastic:9200/log-wlb-security-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 [status:N/A request:20.026s] Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse response.begin() File "/usr/lib64/python3.6/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib64/python3.6/socket.py", line 586, in readinto return self._sock.recv_into(b) socket.timeout: timed ou

aniketpant1 commented 3 years ago

i have set 'es_conn_timeout: 1000' parameter in elastalert conf file but i think it is not working

nsano-rururu commented 3 years ago

i have set 'es_conn_timeout: 1000' parameter in elastalert conf file but i think it is not working

look https://github.com/Yelp/elastalert/issues/2469

Did you restart ElastAlert after adding the settings to config.yaml (if it's a docker container, restart ElastAlert's docker container)

aniketpant1 commented 3 years ago

I kill the process of elastalert and then again start the elastalert elastalert --verbose --config /path/to/config

On Sat, Jan 16, 2021, 9:06 PM Naoyuki Sano notifications@github.com wrote:

i have set 'es_conn_timeout: 1000' parameter in elastalert conf file but i think it is not working

look

2469 https://github.com/Yelp/elastalert/issues/2469

Did you restart ElastAlert after adding the settings to config.yaml (if it's a docker container, restart ElastAlert's docker container)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Yelp/elastalert/issues/3093#issuecomment-761583037, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVJXF6FKISY7KS6FSAXQT3S2GW73ANCNFSM4WDTOIMA .

aniketpant1 commented 3 years ago

"mappings": { "properties": { "match_body": { "properties": { "z_logstash_pipeline": { "type": "keyword" }, "etl_pipeline": { "type": "keyword" } } } } } elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'mapper [match_body.etl_pipeline] cannot be changed from type [long] to [keyword]') i've set the template and delete the old indices and recreate the index by elastalert-create-index and run the elastalert