RicoVZ
commented
4 years ago Hi realforce212,
Thanks for posting an issue and apologies for the late reply.
At the moment, the ES version Cuckoo supports is deprecated. Cuckoo only ships with templates for ES 5, and the code also assumes it can retrieve multiple doctypes from a single index. This behavior is no longer possible in ES. We have not updates the templates (yet).
This PR was made with a fix for the problem: https://github.com/cuckoosandbox/cuckoo/pull/2627. I have not tested it, and cannot say if it is backwards compatible if you currently already are using ES.
tincho9
vinceplayer
anust
jgru
My issue is:Cuckoo Elastic Search not working
My Cuckoo version and operating system are:
The cuckoo version is the last version of today and operating system is ubuntu 18
This can be reproduced by: enabling elastic search in reporting.conf and run cuckoo -d command
The log, error, files etc can be found at:
2019-12-01 19:34:56,528 [cuckoo.core.database] DEBUG: Using database-wide lock for sqlite 2019-12-01 19:34:56,568 [cuckoo.core.startup] DEBUG: Imported modules... 2019-12-01 19:34:56,753 [elasticsearch] WARNING: PUT http://127.0.0.1:9200/_template/cuckoo_template [status:400 request:0.076s] Oops! Cuckoo failed in an unhandled exception! Sometimes bugs are already fixed in the development release, it is therefore recommended to retry with the latest development release available https://github.com/cuckoosandbox/cuckoo If the error persists please open a new issue at https://github.com/cuckoosandbox/cuckoo/issues
=== Exception details === Cuckoo version: 2.0.7 OS version: posix OS release: Ubuntu 18.04 bionic Python version: 2.7.15+ Python implementation: CPython Machine arch: x86_64 Modules: alembic:1.0.10 androguard:3.0.1 argparse:1.2.1 asn1crypto:0.24.0 attrs:19.1.0 backports-abc:0.5 backports.shutil-get-terminal-size:1.0.0 beautifulsoup4:4.5.3 bleach:3.1.0 bottle:0.12.13 capstone:3.0.5rc2 cffi:1.12.2 chardet:2.3.0 click:6.6 colorama:0.3.7 configparser:3.7.3 cryptography:2.6.1 cuckoo:2.0.7 decorator:4.4.0 defusedxml:0.5.0 distorm3:3.4.1 django-extensions:1.6.7 django:1.8.4 dpkt:1.8.7 dumbnet:1.12 ecdsa:0.13 egghatch:0.2.3 elasticsearch:5.3.0 entrypoints:0.3 enum34:1.1.6 et-xmlfile:1.0.1 flask-sqlalchemy:2.4.0 flask:0.12.2 functools32:3.2.3.post2 future:0.17.1 futures:3.2.0 gevent:1.2.2 greenlet:0.4.15 httpreplay:0.2.4 idna:2.8 ipaddress:1.0.22 ipykernel:4.10.0 ipython-genutils:0.2.0 ipython:5.8.0 ipywidgets:7.4.2 itsdangerous:1.1.0 jdcal:1.4 jinja2:2.9.6 jsbeautifier:1.6.2 jsonschema:3.0.1 jupyter-client:5.2.4 jupyter-console:5.2.0 jupyter-core:4.4.0 jupyter:1.0.0 keyring:10.6.0 keyrings.alt:3.0 libvirt-python:4.0.0 mako:1.0.7 markupsafe:1.1.1 mistune:0.8.4 nbconvert:5.4.1 nbformat:4.4.0 notebook:5.7.6 olefile:0.43 oletools:0.51 openpyxl:2.6.1 pandocfilters:1.4.2 pathlib2:2.3.3 peepdf:0.4.2 pefile2:1.2.11 pefile:2017.11.5 pexpect:4.6.0 pickleshare:0.7.5 pillow:3.2.0 pip:19.3.1 prometheus-client:0.6.0 prompt-toolkit:1.0.15 ptyprocess:0.6.0 pycparser:2.19 pycrypto:2.6.1 pydeep:0.4 pyelftools:0.24 pygments:2.3.1 pygobject:3.26.1 pyguacamole:0.6 pymisp:2.4.106 pymongo:3.0.3 pyopenssl:19.0.0 pyrsistent:0.14.11 python-dateutil:2.4.2 python-editor:1.0.4 python-magic:0.4.12 python:2.7.15- pythonaes:1.0 pytz:2018.3 pyxdg:0.25 pyzmq:18.0.1 qtconsole:4.4.3 requests:2.13.0 roach:0.1.2 scandir:1.10.0 scapy:2.3.2 secretstorage:2.3.1 send2trash:1.5.0 setuptools:41.6.0 sflock:0.3.10 simplegeneric:0.8.1 singledispatch:3.4.0.3 six:1.12.0 sqlalchemy:1.3.3 sqlparse:0.2.4 terminado:0.8.1 testpath:0.4.2 tlslite-ng:0.6.0 tornado:5.1.1 traitlets:4.3.2 ujson:1.35 unicorn:1.0.1 urllib3:1.24.1 virtualenv:15.1.0 volatility:2.6.1 wakeonlan:0.2.2 wcwidth:0.1.7 webencodings:0.5.1 werkzeug:0.14.1 wheel:0.30.0 widgetsnbextension:3.4.2 wsgiref:0.1.2 yara-python:3.6.3
2019-12-01 19:34:56,759 [cuckoo] ERROR: RequestError: TransportError(400, u'mapper_parsing_exception', u'Root mapping definition has unsupported parameters: [call : {dynamic_templates=[{not_analyzed={mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=string, match=}}, {call_arguments={path_match=arguments., mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=long, match=}}], date_detection=false, properties={report_time={format=epoch_second, type=date}}}] [cuckoo : {dynamic_templates=[{not_analyzed={mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=string, match=}}, {signatures={path_match=signatures.marks.call.arguments., path_unmatch=signatures.marks.call.arguments.registers., mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=long, match=}}], date_detection=false, properties={report_time={format=epoch_second, type=date}, procmemory={include_in_root=True, type=nested, properties={regions={include_in_root=True, type=nested}}}}}] [irma : {dynamic_templates=[{notanalyzed={mapping={index=not_analyzed, type=string, doc_values=True}, match_mapping_type=string, match=}}], properties={timestamp_first_scan={format=epoch_millis, type=date}, timestamp_last_scan={format=epoch_millis, type=date}}}]') Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/cuckoo/main.py", line 297, in main cuckoo_init(level, ctx) File "/usr/local/lib/python2.7/dist-packages/cuckoo/main.py", line 190, in cuckoo_init init_modules() File "/usr/local/lib/python2.7/dist-packages/cuckoo/core/startup.py", line 274, in init_modules module.init_once() File "/usr/local/lib/python2.7/dist-packages/cuckoo/reporting/elasticsearch.py", line 50, in init_once if not cls.apply_template(): File "/usr/local/lib/python2.7/dist-packages/cuckoo/reporting/elasticsearch.py", line 75, in apply_template name=cls.template_name, body=json.dumps(template) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped return func(*args, params=params, *kwargs) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/indices.py", line 458, in put_template name), params=params, body=body) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 318, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request self._raise_error(response.status, raw_data) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 124, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) RequestError: TransportError(400, u'mapper_parsing_exception', u'Root mapping definition has unsupported parameters: [call : {dynamic_templates=[{not_analyzed={mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=string, match=}}, {call_arguments={path_match=arguments., mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=long, match=}}], date_detection=false, properties={report_time={format=epoch_second, type=date}}}] [cuckoo : {dynamic_templates=[{not_analyzed={mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=string, match=}}, {signatures={path_match=signatures.marks.call.arguments., path_unmatch=signatures.marks.call.arguments.registers., mapping={ignore_above=32766, index=not_analyzed, type=string}, match_mapping_type=long, match=}}], date_detection=false, properties={report_time={format=epoch_second, type=date}, procmemory={include_in_root=True, type=nested, properties={regions={include_in_root=True, type=nested}}}}}] [irma : {dynamic_templates=[{notanalyzed={mapping={index=not_analyzed, type=string, doc_values=True}, match_mapping_type=string, match=*}}], properties={timestamp_first_scan={format=epoch_millis, type=date}, timestamp_last_scan={format=epoch_millis, type=date}}}]')