kyokoshome
commented
5 years ago Sample VPC log format now:
2 667162227571 eni-1231231231231231 10.3.14.217 1.1.1.1 29800 443 6 9 814 1551076978 1551077038 ACCEPT OK
2 667164967571 eni-1231231231231231 10.3.14.217 2.2.2.2 12191 443 6 21 5692 1551076978 1551077038 ACCEPT OK
2 667164967571 eni-1231231231231231 3.3.3.3 10.3.14.217 443 5949 6 4032 161316 1551076978 1551077038 ACCEPT OK
I am following the guide at Readme.md on MAC, but failed at import log step:
$ python importLogs.py --logtype vpc --logdir /Users/wumark/workspace/temp/vpclog/
Beginning import process Creating mapping in ES for index: vpc_flowlogs Creating Ingest Pipeline for index: vpc_flowlogs Creating new index-pattern in .kibana index Setting formatted fields on index-pattern Setting index-pattern as default index Deleting useless index-patterns in .kibana index Deleting index-pattern: .ml-anomalies- Deleting index-pattern: .ml-notifications importing saved objects into Kibana Begin importing log files File: .DS_Store is not the correct format. File need to end with .gz Importing log file: /Users/wumark/workspace/temp/vpclog//667164967571_vpcflowlogs_ap-northeast-1_fl-09efe29fb030b37b0_20190225T0645Z_9ba3c655.log.gz Traceback (most recent call last): File "importLogs.py", line 356, in
loadFiles()
File "importLogs.py", line 211, in loadFiles
processFiles(f)
File "importLogs.py", line 168, in processFiles
for i in status:
File "/usr/local/aws/lib/python2.7/site-packages/elasticsearch/helpers/init.py", line 306, in parallel_bulk
_chunk_actions(actions, chunk_size, max_chunk_bytes, client.transport.serializer)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 668, in next
raise value
elasticsearch.helpers.BulkIndexError: (u'500 document(s) failed to index.', [{u'index': {u'status': 500, u'_type': u'vpc_flowlogs', u'_index': u'vpc_flowlogs', u'error': {u'caused_by': {u'caused_by': {u'reason': u'Provided Grok expressions do not match field value: [version account-id interface-id srcaddr dstaddr sr (Skip)