problem inserting some tasks with yara results into elasticsearch (search only + mongo)

[mallorybobalice]

hi,

I have ES 'searchonly' + mongo for reporting.

I've also added a bit of code to the reporting module at the bottom to add signatures into the search (btw the code that's there probs needs some if checks)

            report["task_id"] = results["info"]["id"]
            if results.has_key("info") and results["info"] and results["info"].has_key("started"):
                report["taskstart"] = results.get("info",{}).get("started")
            if results.has_key("info") and results["info"] and results["info"].has_key("ended"):
                report["taskend"] = results.get("info",{}).get("ended")
            if results.has_key("info") and results["info"]:
                report["info"]    = results.get("info")
            if results.has_key("target") and results["target"]:
                report["target"]  = results.get("target")
            if results.has_key("behavior") and results["behavior"] and results["behavior"].has_key("summary"):
                report["summary"] = results.get("behavior", {}).get("summary")
            if results.has_key("network") and results["network"]:
                report["network"] = results.get("network")
            if results.has_key("signatures") and results["signatures"]:
                report["signatures"] = results.get("signatures")
            if results.has_key("malscore") and results["malscore"]:
                report["malscore"] = results.get("malscore")
            if results.has_key("malfamily") and results["malfamily"]:
                report["malfamily"] = results.get("malfamily")
            if results.has_key("virustotal") and results["virustotal"]:
                report["virustotal"] = results.get("virustotal")
            if results.has_key("virustotal") and results["virustotal"] and results["virustotal"].has_key("positives") and results["virustotal"].has_key("total"):
                report["virustotal_summary"] = "%s/%s" % (results["virustotal"]["positives"],results["virustotal"]["total"])

for some tasks I get this:

2016-09-27 15:50:31,909 [root] INFO: Task #128044: reports generation completed
--
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 279, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 113, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, u'mapper_parsing_exception', u'failed to parse [target.file.yara.meta.date]')

        "signatures": [], 
        "target": {
            "category": "file", 
            "file": {
                "yara": [
                    {
                        "meta": {
                            "date": "01/21/13", 
                            "filetype": "RTF", 

[mallorybobalice]

any ideas?

I presume it's saying 'i don't like the date format' I can't really guarantee that field would have a valid date format but would be happy to drop it. (guess I could also edit it in all the signatures but that's probably less practical)...

grep date cuckoo/data/yara/////* returns a about as many different date formats as signatures

[mallorybobalice]

hmmm, is anyone familiar enough with ES to help formulate a put statement to disable the mapping?

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

either for signatures.target.file.yara.meta.date or target.file.yara.meta.date. i'm not actually sure of the nesting... but the json doc part is above...

[mallorybobalice]

anyone?

i'll dig into the logs to see what else is flagging in the next few days