search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.65k stars 24.65k forks source link

Cuckoo integration with elastic search 7 mapping issue #73983

Closed iemshubh closed 3 years ago

iemshubh commented 3 years ago

This can be reproduced by: enabling elastic search in reporting.conf and run cuckoo -d command The log, error, files etc can be found at:

2019-12-01 19:34:56,528 [cuckoo.core.database] DEBUG: Using database-wide lock for sqlite 2019-12-01 19:34:56,568 [cuckoo.core.startup] DEBUG: Imported modules... 2019-12-01 19:34:56,753 [elasticsearch] WARNING: PUT http://192.168.1.11:9200/_template/cuckoo_template [status:400 request:0.076s] Oops! Cuckoo failed in an unhandled exception! Sometimes bugs are already fixed in the development release, it is therefore recommended to retry with the latest development release available https://github.com/cuckoosandbox/cuckoo If the error persists please open a new issue at https://github.com/cuckoosandbox/cuckoo/issues

=== Exception details === Cuckoo version: 2.0.7 OS version: posix OS release: Ubuntu 18.04 bionic Python version: 2.7.15+ Python implementation: CPython Machine arch: x86_64 Modules: alembic:1.0.10 androguard:3.0.1 argparse:1.2.1 asn1crypto:0.24.0 attrs:19.1.0 backports-abc:0.5 backports.shutil-get-terminal-size:1.0.0 beautifulsoup4:4.5.3 bleach:3.1.0 bottle:0.12.13 capstone:3.0.5rc2 cffi:1.12.2 chardet:2.3.0 click:6.6 colorama:0.3.7 configparser:3.7.3 cryptography:2.6.1 cuckoo:2.0.7 decorator:4.4.0 defusedxml:0.5.0 distorm3:3.4.1 django-extensions:1.6.7 django:1.8.4 dpkt:1.8.7 dumbnet:1.12 ecdsa:0.13 egghatch:0.2.3 elasticsearch:5.3.0 entrypoints:0.3 enum34:1.1.6 et-xmlfile:1.0.1 flask-sqlalchemy:2.4.0 flask:0.12.2 functools32:3.2.3.post2 future:0.17.1 futures:3.2.0 gevent:1.2.2 greenlet:0.4.15 httpreplay:0.2.4 idna:2.8 ipaddress:1.0.22 ipykernel:4.10.0 ipython-genutils:0.2.0 ipython:5.8.0 ipywidgets:7.4.2 itsdangerous:1.1.0 jdcal:1.4 jinja2:2.9.6 jsbeautifier:1.6.2 jsonschema:3.0.1 jupyter-client:5.2.4 jupyter-console:5.2.0 jupyter-core:4.4.0 jupyter:1.0.0 keyring:10.6.0 keyrings.alt:3.0 libvirt-python:4.0.0 mako:1.0.7 markupsafe:1.1.1 mistune:0.8.4 nbconvert:5.4.1 nbformat:4.4.0 notebook:5.7.6 olefile:0.43 oletools:0.51 openpyxl:2.6.1 pandocfilters:1.4.2 pathlib2:2.3.3 peepdf:0.4.2 pefile2:1.2.11 pefile:2017.11.5 pexpect:4.6.0 pickleshare:0.7.5 pillow:3.2.0 pip:19.3.1 prometheus-client:0.6.0 prompt-toolkit:1.0.15 ptyprocess:0.6.0 pycparser:2.19 pycrypto:2.6.1 pydeep:0.4 pyelftools:0.24 pygments:2.3.1 pygobject:3.26.1 pyguacamole:0.6 pymisp:2.4.106 pymongo:3.0.3 pyopenssl:19.0.0 pyrsistent:0.14.11 python-dateutil:2.4.2 python-editor:1.0.4 python-magic:0.4.12 python:2.7.15- pythonaes:1.0 pytz:2018.3 pyxdg:0.25 pyzmq:18.0.1 qtconsole:4.4.3 requests:2.13.0 roach:0.1.2 scandir:1.10.0 scapy:2.3.2 secretstorage:2.3.1 send2trash:1.5.0 setuptools:41.6.0 sflock:0.3.10 simplegeneric:0.8.1 singledispatch:3.4.0.3 six:1.12.0 sqlalchemy:1.3.3 sqlparse:0.2.4 terminado:0.8.1 testpath:0.4.2 tlslite-ng:0.6.0 tornado:5.1.1 traitlets:4.3.2 ujson:1.35 unicorn:1.0.1 urllib3:1.24.1 virtualenv:15.1.0 volatility:2.6.1 wakeonlan:0.2.2 wcwidth:0.1.7 webencodings:0.5.1 werkzeug:0.14.1 wheel:0.30.0 widgetsnbextension:3.4.2 wsgiref:0.1.2 yara-python:3.6.3

2019-12-01 19:34:56,759 [cuckoo] ERROR: RequestError: TransportError(400, u'mapper_parsing_exception', u'Root mapping definition has unsupported parameters: [call : {dynamic_templates=[{not_analyzed={mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=string, match=}}, {call_arguments={path_match=arguments., mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=long, match=}}], date_detection=false, properties={report_time={format=epoch_second, type=date}}}] [cuckoo : {dynamic_templates=[{not_analyzed={mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=string, match=}}, {signatures={path_match=signatures.marks.call.arguments., path_unmatch=signatures.marks.call.arguments.registers., mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=long, match=}}], date_detection=false, properties={report_time={format=epoch_second, type=date}, procmemory={include_in_root=True, type=nested, properties={regions={include_in_root=True, type=nested}}}}}] [irma : {dynamic_templates=[{notanalyzed={mapping={index=not_analyzed, type=string, doc_values=True}, match_mapping_type=string, match=}}], properties={timestamp_first_scan={format=epoch_millis, type=date}, timestamp_last_scan={format=epoch_millis, type=date}}}]') Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/cuckoo/main.py", line 297, in main cuckoo_init(level, ctx) File "/usr/local/lib/python2.7/dist-packages/cuckoo/main.py", line 190, in cuckoo_init init_modules() File "/usr/local/lib/python2.7/dist-packages/cuckoo/core/startup.py", line 274, in init_modules module.init_once() File "/usr/local/lib/python2.7/dist-packages/cuckoo/reporting/elasticsearch.py", line 50, in init_once if not cls.apply_template(): File "/usr/local/lib/python2.7/dist-packages/cuckoo/reporting/elasticsearch.py", line 75, in apply_template name=cls.template_name, body=json.dumps(template) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped return func(args, params=params, *kwargs) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/indices.py", line 458, in put_template name), params=params, body=body) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 318, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request self._raise_error(response.status, raw_data) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 124, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) RequestError: TransportError(400, u'mapper_parsing_exception', u'Root mapping definition has unsupported parameters: [call : {dynamic_templates=[{not_analyzed={mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=string, match=}}, {call_arguments={path_match=arguments., mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=long, match=}}], date_detection=false, properties={report_time={format=epoch_second, type=date}}}] [cuckoo : {dynamic_templates=[{not_analyzed={mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=string, match=}}, {signatures={path_match=signatures.marks.call.arguments., path_unmatch=signatures.marks.call.arguments.registers., mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=long, match=}}], date_detection=false, properties={report_time={format=epoch_second, type=date}, procmemory={include_in_root=True, type=nested, properties={regions={include_in_root=True, type=nested}}}}}] [irma : {dynamic_templates=[{notanalyzed={mapping={index=not_analyzed, type=string, doc_values=True}, match_mapping_type=string, match=}}], properties={timestamp_first_scan={format=epoch_millis, type=date}, timestamp_last_scan={format=epoch_millis, type=date}}}]')

danielmitterdorfer commented 3 years ago

Thanks for your report! However, this points to an issue with Cuckoo rather than Elasticsearch. I quickly checked the Cuckoo repo and their Github project is archived so I fear it's unlikely the Cuckoo will address this. It may be possible that the version of Elasticsearch does not match Cuckoo's requirements so I suggest you check that you are running with a compatible version.

In any case, as this is not an issue that we can address as Elasticsearch team, I hope you don't mind that I close this.