Handle the output of a python plugin with Malice

[pielco11]

Output of go version:

go version go1.7.4 linux/amd64

Output of docker version:

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:17:43 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:17:43 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 54
 Running: 1
 Paused: 0
 Stopped: 53
Images: 190
Server Version: 17.03.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 490
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.10.0-28-generic
Operating System: Ubuntu 17.04
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.747 GiB
Name: notebook
ID: ZFKE:ULQR:BKKX:3TH7:TV6Z:NDAX:52YB:OMXP:BOWS:UUKT:FM2R:UNAE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: pielco11
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, Docker For Mac, Docker Toolbox, docker-machine, etc.):

I'm running Linux notebook 4.10.0-28-generic #32-Ubuntu SMP Fri Jun 30 05:32:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce the issue:

1. malice scan /path/to/malware.exe

Describe the results you received: ssma: error: argument -f/--filename: expected one argument

Describe the results you expected: No error, actually I don't know what could be a correct output since this is a new plugin.

plugins.toml:

- - - 
[[plugin]]
  name = "ssma"
  enabled = true
  category = "av"
  description = "Simple Static Malware Analysis"
  image = "pielco11/ssma:latest"
  repository = "https://github.com/pielco11/SSMA.git"
  build = false
  mime = "*"
- - - 

Additional information you deem important (e.g. issue happens only occasionally): ssma is a python script that do a static analysis over PE and ELF, this works like ssma [OPTIONS] -f /path/to/malware.exe. As is possible to verify, the Docker image runs as follow ssma -r elasticsearch -f (this is the entrypoint). So what I need to do is docker run --rm -v /path/to/pe.exe:/malware/pe.exe pielco11/ssma pe.exe If I run the plugin alone, it shows some error concerning a connection error to Elasticsearch, even if I firstly run malice elk.

Since the error is about the number of the arguments that I pass to ssma, how can I correctly parse them?

[blacktop]

I'll try to break this in the the parts that might be the issue.

First with respects to elasticsearch this might be because you need to tell elasticsearch inside your plugin to try and connect to elasticsearch here http://elasticsearch:9200 this is because docker creates a DNS entry inside your container that links to the malice elk instance.

You can see that I pull the --link from the config here: https://github.com/maliceio/malice/blob/master/config/config.toml#L48

In my golang based plugins I talk to elasticsearch like so:

elasticsearch.WritePluginResultsToDatabase(elasticsearch.PluginResults{
    ID:       utils.Getopt("MALICE_SCANID", utils.GetSHA256(path)),
    Name:     name,
    Category: category,
    Data:     structs.Map(avira.Results),
})

you can see I try and check for an environment variable passed in by malice of the document_id it created for the scan. That will always be there if you run with malice. If it is NOT there I use the files sha256 instead. This is what I use if I am using a plugin in stand-alone mode without malice.

I also pass in the name ssma and the category av. Data would be the JSON your plugin generates.

So you would do something like this:

es = Elasticsearch(["elasticsearch"])
res = es.index(index="malice", doc_type='sample', id=os.environ.get('MALICE_SCANID',sha256(args.filename), body=rDump)

[blacktop]

I think the other issue is that malce expects that the only arg it will supply to a plugin is the filename SSMA might need the file's full path? /malware/file you might want to add a check to your plugin to make sure that the file exists that was passed in for arg.filename if it doesn't exist maybe it is running in a different place then the file and you need to pass it the full path to the file?

In my golang plugins I do that as follows:

path, err = filepath.Abs(c.Args().First())
            utils.Assert(err)

            if _, err := os.Stat(path); os.IsNotExist(err) {
                utils.Assert(err)
            }

I get the full path no matter what and then I check to make sure that files exists inside the container

The only other thing I can think of is that malice usually runs plugins as the malice user and maybe it isn't seeing the file because the user you are running as doesn't have permissions to touch the file?

[pielco11]

Breaking down the code, I see that the error is given by a -t argument that is not handled correctly. In particular, if I set filename as positional argument and not optional, I get ssma: error: unrecognized arguments: -t. So if I set filename as optional, what ssma sees (at least, what makes sense, to me, that ssma is probably seeing) is something like ssma -r elasticsearch -f -t /full/path/to/malware.exe.

Now, I set filename as positional and add -t as optional argument, the error that I get when I run malice scan /full/path/to/malware.exe is ssma: error: the following arguments are required: filename. So, there could be a problem while parsing arguments (filename in particular) to my plugin.

I build the image with another tag noel(the difference is that ssma will not post to elasticsearch, nothing more) it (my plugin, ssma) alone works great when I execute docker run --rm -v /home/user/final.exe:/malware/final.exe:ro pielco11/ssma:noel final.exe.

When can I do? I'm quite confused.

[blacktop]

Ah yes! I apologize, I am grateful you are working through this with me as it is bringing to light all the details I need to add to documentation for others to make plugins.

Another requirement for a malice plugin is that is can output as JSON and Markdown tables. Is there a way for you to add a flag to SSMA that will tell it to not output JSON to the command line, but instead a Markdown table/representation of the SSMA results?

The default way malice wants to run any plugin is

$ docker run -v `pwd`:/malware --link malice-elastic:elasticsearch pielco11/ssma:noel -t final.exe

I suppose this requires you to change the -f to instead be a positional argument that is just the last arg. I believe this is easy in python.

Then you can run SSMA plugin stand-alone like this:

JSON output to terminal (plus index into elasticsearch)

NOTE: no -t flag

$ docker run -v `pwd`:/malware --link malice-elastic:elasticsearch pielco11/ssma:noel final.exe

=OR=

Markdown output to terminal (plus try to index into elasticsearch)

$ docker run -v `pwd`:/malware --link malice-elastic:elasticsearch pielco11/ssma:noel -t final.exe

In all my plugins I always try to index into elasticsearch and fail silently if there is no elasticsearch node accessible.

You could do this by wrapping your post to elasticsearch in a try/except that just passes on exceptions?

[pielco11]

We are getting closer to the solution, but now I'm getting this error

PUT http://elasticsearch:9200/malice/sample/AV2eMNP-mgtqsE59-_NT [status:400 request:0.874s]

.

So I suppose that the problem is that AV2eMNP-mgtqsE59-_NT should be 05144140e110798339835d3e7176800179c20de8993d78b9488990aa636b5750 because looking at the output of malice, at a point I get time="2017-08-01T14:26:47Z" level=fatal msg="exit status 2" category=av path="/malware/05144140e110798339835d3e7176800179c20de8993d78b9488990aa636b5750" plugin=avast. Believing that this is the reason, this is the code that I use to generate sha256(args.filename) :

hasher = hashlib.sha256()
fl = args.filename
hasher.update(fl.encode('utf-8'))
hashFile = hasher.hexdigest()

But, since AV2eMNP-mgtqsE59-_NT does not change if args.filename = /home/user/final.exe or args.filename = final.exe I don't think that this is the right cause of the problem.

Also I tried some sha256/hexdigest/stuff-like-this and I can't reproduce both AV2eMNP-mgtqsE59-_NT and 05144140e110798339835d3e7176800179c20de8993d78b9488990aa636b5750, don't know how to proceed.

Last but not least, try/except does not work with

es = Elasticsearch(["elasticsearch"])
res = es.index(index="malice", doc_type='sample', id=os.environ.get('MALICE_SCANID',sha256(args.filename), body=rDump)

I'm working for a dirty way to handle this, but now as now at the output there are no errors, so I think that this is a minor problem. Before to try the plugin alone, I prefer to solve these/this problems/problem, just to not overload and keeping the issue as linear as possible, I mean to not shuffle these problems.

[blacktop]

AV2eMNP-mgtqsE59-_NT is most likely the MALICE_SCANID. so when malice started scanning final.exe it auto-generated the document _idand you should use it to append the SSMA results to the rest of the malice plugins results.

[blacktop]

I don't understand why you got a 400 when you tried to PUT to it from python with the es.index command. I think perhaps you need to use an update command and not a create command because you are appending to a document that already exists https://stackoverflow.com/a/30598673

[blacktop]

just FYI the reason why I said you should use either the MALICE_SCANID or the sha256 is because if you scan with malice it will automatically push that env var into the docker container. However, if you are running SSMA stand alone the sha256 is a good unique key to use for the elasticsearch DB because you would only want one record per file. So you can say not scan a file again if it has already been scanned unless the user really wants to

[pielco11]

Now it works perfectly! But if I run it alone, I get some error can not resolve http://elasticsearch.com:9200...

[blacktop]

so this is where we need to decide how to handle es connection failures. Do we ignore them? Do we try a few different hosts?

What I do is I try:

• http://elasticsearch:9200
• MALICE_ELASTICSEARCH
• http://127.0.0.1:9200

if none of those are found then the user doesn't have/want to use elasticsearch

[blacktop]

so a way to run SSMA alone and have it use some independent elasticsearch you would do something like: https://github.com/malice-plugins/yara/blob/master/docs/elasticsearch.md

$ docker volume create --name malice
$ docker run -d --name elasticsearch \
                -p 9200:9200 \
                -v malice:/usr/share/elasticsearch/data \
                 blacktop/elasticsearch
$ docker run --rm -v /path/to/malware:/malware:ro --link elasticsearch pielco11/ssma -t FILE

OR if you wanted to use the malice-elk you would so this:

$ docker run --rm -v /path/to/malware:/malware:ro --link malice-elastic:elasticsearch pielco11/ssma -t FILE

NOTE: I am linking the malice elasticsearch into your plugin AS the host elasticsearch

OR you could have some main elasticsearch cluster somewhere you wanted to use then you would do:

$ docker run --rm -v /path/to/malware:/malware:ro -e MALICE_ELASTICSEARCH=http://pielco11.com:9200 pielco11/ssma -t FILE

NOTE: that sets an environment var inside the plugin to tell ssma where the elasticsearch host is that it should use

[pielco11]

Now with

es = Elasticsearch(["elasticsearch", "127.0.0.1", os.environ.get("MALICE_ELASTICSEARCH")])

(if I understood correctly, MALICE_ELASTICSEARCH is an env var) it works perfectly and no error is shown. So I think that the solution is specifing multiple hosts to point to and linking malice-elastic:elasticsearch.

[blacktop]

that is a great solution

you might still want to catch the exception if none of those are available and silently fail (and do not try to index) meaning the user doesn't wish to write to a DB and just wants to see the results in the terminal.

[pielco11]

Ok, now I have a markdown table and the errors are ignored so the output is pretty clean.

[blacktop]

can you post an example of the Markdown output here? I'd like to see how it turned out 👍

[pielco11]

#### SSMA
| Sections      | Functions   |
|:-------------:|:-----------:|
| 8 | 10 |

Maybe in the future it will change, but now as now it's like this.

EDIT: I'm planning to add the entropy of every section, hostnames, email addresses, etc...

[blacktop]

you don't have to codify it (the markdown) it is meant to be displayed on github ;)

SSMA

Sections	Functions
8	10

---

looks good 👍

[pielco11]

Can now we assume that the plugin is completed? Or is there something more to do?

[blacktop]

I believe it is done. I will hopefully work this weekend to make it easier for people to install community plugins as right now they would have to manually add it to the plugin.toml file. We might need to have people add a config.toml or settings.toml file in the root of the plugin that malice could read/parse when installing

[blacktop]

Here is the awesome plugin - https://github.com/pielco11/SSMA