Closed ghost closed 7 years ago
No, this is not a feature right now.
An interesting idea. That would allow rules to share portions of what they match. I can see value in this. But formatting of the file would have be agreed upon.
Need this feature too. ASAP. :) Will be really useful to handle long list of malicious domains, URLs, IPs
I did a small modification to allow for this (still testing before submitting a pull request) ... it works for both blacklist and whitelist. This will do two things:
- "!file /path/to/list"
set
instead of a list
in the blacklist/whitelist entries, which is much faster with larger sets and deduplicates automatically.
For example:
blacklist:
You can mix between the list entry definition formats, or use either one.
Feedback is welcome :)
please find it here: https://github.com/SherifEldeeb/elastalert/commit/f3df37a052e55f246e9f402bd5a64ecd866969a0
Thank you very much!
@pushkarevds @SherifEldeeb
I would like to use the above method to whitelist IP address and specific string. Currently, I have a cardinality rule and I would like to do whitelist on few strings and IP address by combining the rule as shown below. The rule runs without any error but it's not working as expected. Can someone assist me is this a right method to combine the rule for whitelist?
Main Rule
type: cardinality
filter:
- terms:
rule_name:
- whitelist-zone --> Rule name for whitelisting zone
- whitelisti-ip --> Rule name for whitelisting IP address
whitelist-zone
root@ubuntu:/tmp/elastalert# more whitelist-zone.yaml
# ElastAlert Rule Name
name: whitelist-zone
# ElasticSearch host and port details
es_host: 192.168.96.141
es_port: 9200
# ElasticSearch Index Name
index: graylog_0
# Timestamp added to overcome ES graylog parsing issue
timestamp_field: timestamp
timestamp_type: custom
timestamp_format: '%Y-%m-%d %H:%M:%S.%f'
timestamp_format_expr: 'ts[:23] + ts[26:]'
doc_type: message
#writeback_index: elastalert_status
# Network Port scan aggregation Rule 50 counts in 60 Seconds
type: whitelist
compare_key: srx-source-zone-name
whitelist:
- "!file /opt/whitelist-zone.txt"
whitelisti-ip
root@ubuntu:/tmp/elastalert# more whitelist-ip.yaml
# ElastAlert Rule Name
name: whitelisti-ip
# ElasticSearch host and port details
es_host: 192.168.96.141
es_port: 9200
# ElasticSearch Index Name
index: graylog_0
# Timestamp added to overcome ES graylog parsing issue
timestamp_field: timestamp
timestamp_type: custom
timestamp_format: '%Y-%m-%d %H:%M:%S.%f'
timestamp_format_expr: 'ts[:23] + ts[26:]'
doc_type: message
#writeback_index: elastalert_status
# Network Port scan aggregation Rule 50 counts in 60 Seconds
type: whitelist
compare_key: srx-source-address
whitelist:
- "!file /opt/whitelist-IP.txt"
Files
root@ubuntu:/tmp/elastalert# cat /opt/whitelist-IP.txt
10.18.109.59
10.10.10.10
20.20.20.20
30.30.30.30
root@ubuntu:/tmp/elastalert# cat /opt/whitelist-zone.txt
EXTERNAL
Internet
root@ubuntu:/tmp/elastalert#
Maybe I don't know if this feature, but you have "-->" in rule names. Do you think that is causing problems. In yaml you use # to start a comment. Correct me if I'm wrong.
On Fri, Jul 14, 2017, 5:50 AM sathishdsgithub notifications@github.com wrote:
@pushkarevds https://github.com/pushkarevds @SherifEldeeb https://github.com/sherifeldeeb
I would like to use the above method to whitelist IP address and specific string. Currently, I have a cardinality rule and I would like to do whitelist on few strings and IP address by combining the rule as shown below. The rule runs without any error but it's not working as expected. Can someone assist me is this a right method to combine the rule for whitelist?
Main Rule
type: cardinality filter:
- terms: rule_name:
- whitelist-zone --> Rule name for whitelisting zone
- whitelisti-ip --> Rule name for whitelisting IP address
whitelist-zone
root@ubuntu:/tmp/elastalert# more whitelist-zone.yaml
ElastAlert Rule Name
name: whitelist-zone
ElasticSearch host and port details
es_host: 192.168.96.141
es_port: 9200
ElasticSearch Index Name
index: graylog_0
Timestamp added to overcome ES graylog parsing issue
timestamp_field: timestamp timestamp_type: custom timestamp_format: '%Y-%m-%d %H:%M:%S.%f' timestamp_format_expr: 'ts[:23] + ts[26:]'
doc_type: message
writeback_index: elastalert_status
Network Port scan aggregation Rule 50 counts in 60 Seconds
type: whitelist
compare_key: srx-source-zone-name
whitelist:
- "!file /opt/whitelist-zone.txt"
whitelisti-ip
root@ubuntu:/tmp/elastalert# more whitelist-ip.yaml
ElastAlert Rule Name
name: whitelisti-ip
ElasticSearch host and port details
es_host: 192.168.96.141
es_port: 9200
ElasticSearch Index Name
index: graylog_0
Timestamp added to overcome ES graylog parsing issue
timestamp_field: timestamp timestamp_type: custom timestamp_format: '%Y-%m-%d %H:%M:%S.%f' timestamp_format_expr: 'ts[:23] + ts[26:]'
doc_type: message
writeback_index: elastalert_status
Network Port scan aggregation Rule 50 counts in 60 Seconds
type: whitelist
compare_key: srx-source-address
whitelist:
- "!file /opt/whitelist-IP.txt"
Files
root@ubuntu:/tmp/elastalert# cat /opt/whitelist-IP.txt 10.18.109.59 10.10.10.10 20.20.20.20 30.30.30.30 root@ubuntu:/tmp/elastalert# cat /opt/whitelist-zone.txt EXTERNAL Internet root@ubuntu:/tmp/elastalert#
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Yelp/elastalert/issues/711#issuecomment-315351632, or mute the thread https://github.com/notifications/unsubscribe-auth/AB9SJstl1tw8nwGk76J9LFTDHDrFZjWyks5sN2QKgaJpZM4J_FP8 .
@bean5 I have just mentioned that for understanding ,My actual rule doesn't have that
Okay. Hopefully someone else more experienced can locate your real issue.
On Fri, Jul 14, 2017, 8:10 AM sathishdsgithub notifications@github.com wrote:
@bean5 https://github.com/bean5 I have just mentioned that for understanding ,My actual rule doesn't have that
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/Yelp/elastalert/issues/711#issuecomment-315384693, or mute the thread https://github.com/notifications/unsubscribe-auth/AB9SJt9PmjsYLFVYcqWtj-Qq93Rkmb3Uks5sN4T8gaJpZM4J_FP8 .
Include blacklist and whitelist using files rule blacklist:
Will it be realized?