search

alertmanager / alert_manager

Splunk Alert Manager with advanced reporting on alerts, workflows (modify assignee, status, severity) and auto-resolve features
Other
81 stars 43 forks source link

some alert will not insert into alert index #291

Open fist-xp opened 3 years ago

fist-xp commented 3 years ago

some alert will not insert into alert index. I found some err info in _internal index.

8/5/214:10:01.165 PM 08-05-2021 16:10:01.165 +0800 WARN sendmodalert - action=alert_manager - Alert action script returned error code=1host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
  8/5/214:10:01.165 PM 08-05-2021 16:10:01.165 +0800 INFO sendmodalert - action=alert_manager - Alert action script completed in duration=666 ms with exit code=1host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
  8/5/214:10:01.120 PM 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - UnicodeEncodeError: 'latin-1' codec can't encode characters in position 160-161: Body ('天空') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
  8/5/214:10:01.120 PM 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - (name.title(), data[err.start:err.end], name)) from Nonehost = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
  8/5/214:10:01.120 PM 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - File "/data/Splunk/splunk/lib/python3.7/http/client.py", line 170, in _encodehost = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
  8/5/214:10:01.120 PM 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - body = _encode(body, 'body')host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
  8/5/214:10:01.120 PM 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - File "/data/Splunk/splunk/lib/python3.7/http/client.py", line 1289, in _send_requesthost = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
  8/5/214:10:01.120 PM 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - self._send_request(method, url, body, headers, encode_chunked)host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
fist-xp commented 3 years ago

8/5/214:10:01.165 PM | 08-05-2021 16:10:01.165 +0800 WARN sendmodalert - action=alert_manager - Alert action script returned error code=1host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

8/5/214:10:01.165 PM | 08-05-2021 16:10:01.165 +0800 INFO sendmodalert - action=alert_manager - Alert action script completed in duration=666 ms with exit code=1host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd   8/5/214:10:01.120 PM | 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - UnicodeEncodeError: 'latin-1' codec can't encode characters in position 160-161: Body ('天空') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

8/5/214:10:01.120 PM | 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - (name.title(), data[err.start:err.end], name)) from Nonehost = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

8/5/214:10:01.120 PM | 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - File "/data/Splunk/splunk/lib/python3.7/http/client.py", line 170, in _encodehost = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

8/5/214:10:01.120 PM | 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - body = _encode(body, 'body')host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

8/5/214:10:01.120 PM | 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - File "/data/Splunk/splunk/lib/python3.7/http/client.py", line 1289, in _send_requesthost = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

8/5/214:10:01.120 PM | 08-05-2021 16:10:01.120 +0800 ERROR sendmodalert - action=alert_manager STDERR - self._send_request(method, url, body, headers, encode_chunked)host = bj-vm-sec-searchhead-splunk-188source = /data/Splunk/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

fist-xp commented 3 years ago

at the same time i have a email alert, this is normal, only the alert manager 's alert not generate

fist-xp commented 3 years ago

Some other sample log, it seems like alert manager not support Chinese charater.

the Chinese charater were contained in splunk alert name, not alert manager alert title ,because I change all title in English.

8/6/218:10:02.402 AM | 08-06-2021 08:10:02.402 +0800 ERROR sendmodalert - action=alert_manager STDERR - UnicodeEncodeError: 'latin-1' codec can't encode characters in position 171-177: Body ('文件完整性告警') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.host = bj-vm-sec-searchhead-splunk-188index = _internalsourcetype = splunkdsplunk_server = bj-vm-sec-searchhead-splunk-188

8/6/218:10:02.319 AM | 2021-08-06 08:10:02,319 INFO pid="86180" logger="alert_manager_suppression_helper" message="Checking for matching suppression rules for alert=/etc/passwd文件完整性告警" (SuppressionHelper.py:66)host = bj-vm-sec-searchhead-splunk-188index = _internalmessage = Checking for matching suppression rules for alert=/etc/passwd文件完整性告警sourcetype = alert_manager_suppression_helper-too_smallsplunk_server = bj-vm-sec-searchhead-splunk-188

8/6/218:10:02.248 AM | 2021-08-06 08:10:02,248 INFO pid="86180" logger="alert_manager" message="Found job for alert '/etc/passwd文件完整性告警' with title 'HIDS passwd file monitorning'. Context is 'HIDS_all' with 1 results." (alert_manager.py:566)host = bj-vm-sec-searchhead-splunk-188index = _internalmessage = Found job for alert '/etc/passwd文件完整性告警' with title 'HIDS passwd file monitorning'. Context is 'HIDS_all' with 1 results.sourcetype = alert_manager-too_smallsplunk_server = bj-vm-sec-searchhead-splunk-188

8/6/218:10:01.733 AM | 08-06-2021 08:10:01.733 +0800 INFO sendmodalert - Invoking modular alert action=alert_manager for search="/etc/passwd文件完整性告警" sid="scheduler__splunk_SElEU19hbGw__RMD5bbb47a07bc26a359_at_1628208600_360" in app="HIDS_all" owner="splunk" type="saved"

lukasz1992 commented 3 years ago

@fist-xp You need to change

input.submit(event, hostname = socket.gethostname(), sourcetype = 'incident_change', source = 'alert_handler.py', index=index)

to

input.submit(event.encode('utf-8'), hostname = socket.gethostname(), sourcetype = 'incident_change', source = 'alert_handler.py', index=index)

in line 262 in bin/alert_manager.py