Shipping Mordor logs to HELK

[sec-balkan]

root@ttp:/home/pfctpot/mordor/scripts/data-shippers# python3 Mordor-Elastic.py --url http://localhost:9200 inputs apt29_evals_day1_manual_2020-05-01225525.json
Initializing Elasticsearch connection and index...
Traceback (most recent call last):
  File "Mordor-Elastic.py", line 41, in <module>
    "index.mapping.total_fields.limit": 2000
  File "/usr/local/lib/python3.7/dist-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/elasticsearch/client/indices.py", line 124, in create
    "PUT", _make_path(index), params=params, headers=headers, body=body
  File "/usr/local/lib/python3.7/dist-packages/elasticsearch/transport.py", line 415, in perform_request
    raise e
  File "/usr/local/lib/python3.7/dist-packages/elasticsearch/transport.py", line 388, in perform_request
    timeout=timeout,
  File "/usr/local/lib/python3.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 275, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.7/dist-packages/elasticsearch/connection/base.py", line 331, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, 'resource_already_exists_exception', 'index [winlogbeat-mordor/QsS91DnYSw6SeqSQUNG8JQ] already exists')

[blueteambram]

I was getting this error as well, if you go into Kibana and manage your elasticsearch indexes, you can delete that reference its making to winlogbeat-mordor index, after you delete it itll let you run it again. But i was still getting the issue of not being able to see the logs afterwards. Also, im still having issue of not being able to directly parse json files and only compressed files.

[automate-tim]

Hello, I started working with this project similar to you all and found a few features that may help.

1. There is a flag you can add that will make this error go away (it parses the new data under the original index): add either --no-index-creation or -n to your command line arguments

Code from script argparser.add_argument("--no-index-creation", "-n", action="store_false", dest="create_index", help="Don't create index.")

or

2. You can use the -i or --index flags to specify what index name you would like in your command line arguments

Code from script argparser.add_argument("--index", "-i", default="winlogbeat-mordor", help="Target index for data import (%(default)s)")

[sec-balkan]

Yes! I solved with that :)

How I'm unable to see logs ni ten dashboards... I created a index called logs-mordor and the helk dashboards (which take logs*) doesn't show it.

Any help? Sorry for my english.

[automate-tim]

Check out my other comment about changing the Kibana index on the dashboards pages and let me know if that helps: https://github.com/OTRF/mordor/issues/44#issuecomment-859701402