Connection successful but no data retrieved - Suggestions for troubleshooting?

[lrchma]

When launching Flare it appears to successfully connect to ElasticSearch, and doesn't return any results, but the error about not retrieving any data is returned instantly which makes it seem likes it not actually searching (there are gigabytes of data in the indices)

I don't, however, have Bro/Snort/Surricate in this setup, rather I'm using NetFlow data which is normalised to appropriately typed fields in a custom ES index (SIP, DIP, Dport, @timestamp)

At this stage, I can't work out how to troubleshoot this further, so any tips welcomed, please.

I'm running the beacon example as per http://www.austintaylor.io/detect/beaconing/intrusion/detection/system/command/control/flare/elastic/stack/2017/06/10/detect-beaconing-with-flare-elasticsearch-and-intrusion-detection-systems/ and can successfully retrieve this flow data via Kibana in ES.

flare_beacon -c elasticsearch.ini
[INFO] Attempting to connect to elasticsearch...
[SUCCESS] Connected to elasticsearch on localhost:9200
[INFO] Gathering flow data... this may take a while...
ERROR: Elasticsearch did not retrieve any data. Please ensure your settings are correct inside the config file.

ElasticSearch.ini (I have hacked around the values for min values and timeouts, to no avail)

[beacon]
es_host=localhost
es_index=custom-*
es_port=9200
es_timeout=200000
min_occur=5
min_interval=30
min_percent=1
window=15
threads=2
period=48
kibana_version=5
verbose=true

#Elasticsearch fields for beaconing
field_source_ip=SIP
field_destination_ip=DIP
field_destination_port=Dport
field_timestamp=@timestamp
field_flow_bytes_toserver=BytesOut
field_flow_id=CollectionSequence

#Authentication
username=''
password=''

#set to false if using suricata defaults if you have custom fields
suricata_defaults = false

Running ES "5.5.0" on CentOS7.

python -V
Python 2.7.9

pip list
argparse (1.2.1)
backport-ipaddress (0.1)
cffi (0.8.6)
chardet (2.3.0)
colorama (0.3.2)
configparser (3.5.0)
cryptography (0.6.1)
elasticsearch (5.4.0)
Flare (0.3)
gyp (0.1)
html5lib (0.999)
idna (2.6)
ipaddr (2.1.11)
lxml (3.4.0)
MarkupSafe (0.23)
ndg-httpsclient (0.3.2)
numpy (1.13.1)
pandas (0.20.3)
pip (1.5.6)
ply (3.4)
pyasn (1.5.0b7)
pyasn1 (0.1.7)
pycparser (2.10)
pyOpenSSL (0.14)
python-dateutil (2.6.1)
python-geoip (1.2)
python-geoip-geolite2 (2015.0303)
pytz (2017.2)
requests (2.4.3)
requests-file (1.4.2)
scikit-learn (0.19.0)
scipy (0.19.1)
setuptools (5.5.1)
six (1.8.0)
sklearn (0.0)
tldextract (2.0.1)
urllib3 (1.9.1)
wheel (0.24.0)
wsgiref (0.1.2)

Exemplary toolkit here btw :)

Cheers,

Chris

[austin-taylor]

Hello, Chris

Please pull the latest version and try it again. I identified an issue where the check was being set to logstash-flow. Also, do you have a field named event_type:flow ?

[lrchma]

Thank you for prompt response.

The event_type:flow seems likely cause. I don't have that in my mapping but will test this week creating that field for any flows records received.

I pulled the latest version but same error as above, but that makes sense. I'll aim to update schema and try again this week and let know.

Best Regards,

Chris

[austin-taylor]

If you don't mind sharing your query, I can try to find commonality between the two and restructure the way queries are made.

[SMAPPER]

Closing due to no activity

[HawaYann]

Confirmed that in similar testing environment, issue was the hard-coded event_type name. Last version fixed it.

Just make sure to set the correct event_type name in the config file. In my case, mine was : event_key=event.type