malice-plugins / nsrl

Malice NSRL Plugin
MIT License
7 stars 6 forks source link

Results to elastic issue #3

Closed mwilco03 closed 6 years ago

mwilco03 commented 6 years ago

Command issued: nsrl --elasitcsearch [$PRIVATE_ELASTICSEARCH] -V lookup AABCA0896728846A9D5B841617EBE746

Results Output:

DEBU[0000] Number of lines in NSRLFile.txt: 36061377
DEBU[0000] Mapping: {
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true
      }
    },
    "samples": {
      "properties": {
        "file": {
          "properties": {
            "md5": {
              "type": "string"
            },
            "mime": {
              "type": "string"
            },
            "name": {
              "type": "string"
            },
            "path": {
              "type": "string"
            },
            "sha1": {
              "type": "string"
            },
            "sha256": {
              "type": "string"
            },
            "sha512": {
              "type": "string"
            },
            "size": {
              "type": "string"
            }
          }
        },
        "plugins": {
          "properties": {
            "archive": {
              "type": "object"
            },
            "av": {
              "type": "object"
            },
            "document": {
              "type": "object"
            },
            "exe": {
              "type": "object"
            },
            "intel": {
              "type": "object",
              "properties": {
                "virustotal": {
                  "dynamic": false,
                  "type": "object"
                }
              }
            },
            "metadata": {
              "type": "object"
            }
          }
        },
        "scan_date": {
          "type": "date"
        }
      }
    }
  }
}
DEBU[0000] elastic: Error 404 (Not Found): no such index [type=index_not_found_exception]
{"nsrl":{"found":true}}

I decided to attempt to create the index from scratch.

First issue I ran into was _all has been deprecated since 6.0, which would remove the use of default which has been deprecated as well.

Then found out type "string" has been deprecated. Ref: https://www.elastic.co/blog/strings-are-dead-long-live-strings

Changing type "string" to type "text" and removing "default" : { "_all": { "enabled": true }} allowed me to successfully create the index.

So I reissued the command.

Command issued: nsrl --elasitcsearch [$PRIVATE_ELASTICSEARCH] -V lookup AABCA0896728846A9D5B841617EBE746

Results Output: DEBU[0000] Number of lines in NSRLFile.txt: 36061377 DEBU[0000] Index malice already exists. DEBU[0000] elastic: Error 404 (Not Found) {"nsrl":{"found":true}}

After which I'm wondering if the issue may be upstream with malice-plugins/go-plugin-utils/database/elasticsearch/elasticsearch.go

blacktop commented 6 years ago

That is very strange because the mappings have been updated: https://github.com/malice-plugins/go-plugin-utils/blob/master/database/elasticsearch/mapping.go

Can you do a docker pull malice/nsrl to grab latest and see if that helps?

mwilco03 commented 6 years ago

I'll test first thing in the morning. IIRC I pulled from :MD5.

On Wed, Aug 8, 2018 at 10:00 PM blacktop notifications@github.com wrote:

That is very strange because the mappings have been updated: https://github.com/malice-plugins/go-plugin-utils/blob/master/database/elasticsearch/mapping.go

Can you do a docker pull malice/nsrl to grab latest and see if that helps?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/malice-plugins/nsrl/issues/3#issuecomment-411612843, or mute the thread https://github.com/notifications/unsubscribe-auth/AikaKfwhlMPYYgqdf_Q0aw59j8yA70V_ks5uO5fTgaJpZM4V03Pf .

blacktop commented 6 years ago

Hmm maybe the md5 tag didn't finish building in docker cloud. I will check. Thanks

mwilco03 commented 6 years ago

Confirmed. SHA1 Worked. Pulling latest fixed that issue.

blacktop commented 6 years ago

last commit should hopefully fix this

blacktop commented 6 years ago

this is now fixed for sha1 AND md5