Plugins failure - Githubissues

fbarre96 commented 7 years ago

Output of go version:

go version go1.7.1 linux/amd64

Output of docker version:

Client:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:44:08 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:44:08 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 38
 Running: 1
 Paused: 0
 Stopped: 37
Images: 21
Server Version: 1.13.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 224
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.597 GiB
Name: debian
ID: 6G3A:VAFC:U2R7:T7YQ:MXFG:JFPV:RJQC:WT74:ZUWF:EI7P:RZEB:CWR2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, Docker For Mac, Docker Toolbox, docker-machine, etc.): I'm running on a real machine, quite slow but quick enough for my needs.

Steps to reproduce the issue:

Just malice scan SAMPLE after clean install

Describe the results you received: Sorted in time, I have all those plugins that fail. Avast, totalhash,Exiftools, NSRL, avg

File

Field	Value
Name	malware.exe
Path	Téléchargements/malware.exe
Size	5.993 kB
MD5	65bb7a968098bb6b3d62e7edf7cdae39
SHA1	012d4de9f1439348d89dae0e3a2d1ddaf33f31ac
SHA256	b9bfb323d15ad4669781cb93e3c8f01fd2ad37b60d77c43fbe57b0942fbc0598

NSRL Database

Not Found :grey_question:
ShadowServer
Not found

2017/02/07 13:47:52 cannot open; magic mime db is already open

Comodo

Infected	Result	Engine	Updated
true	Malware	1.1	20170129

panic: runtime error: index out of range

goroutine 1 [running]: panic(0x85cde0, 0xc420014090) /usr/local/go/src/runtime/panic.go:500 +0x1a1 main.ParseAvastOutput(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/maliceio/malice-avast/scan.go:119 +0x575 main.AvScan(0x78, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/maliceio/malice-avast/scan.go:84 +0x2b9 main.main.func3(0xc42008a780, 0x0, 0x0) /go/src/github.com/maliceio/malice-avast/scan.go:318 +0x17f github.com/urfave/cli.HandleAction(0x83fe00, 0xc4200154a0, 0xc42008a780, 0x0, 0x0) /go/src/github.com/urfave/cli/app.go:485 +0xd4 github.com/urfave/cli.(*App).Run(0xc42007cea0, 0xc42000c330, 0x3, 0x3, 0x0, 0x0) /go/src/github.com/urfave/cli/app.go:259 +0x74f main.main() /go/src/github.com/maliceio/malice-avast/scan.go:357 +0x78f

F-PROT

Infected	Result	Engine	Updated
true	Toothless.873	4.6.5.141	20170129

time="2017-02-07T13:48:19Z" level=fatal msg="Please supply a valid #totalhash user/key with the flags '--user' and '--key'" time="2017-02-07T13:48:19Z" level=fatal msg="Please supply a valid MALICE_VT_API key with the flag '--api'."

Magic

Field	Value
Mime	application/octet-stream
Description	DOS executable (COM)

SSDeep

24:kT5IyR8dK0LhNqB9sIBzHMb5Js1io1fGOwRQ1O4TulnUxm:kRRF0feBjwU1HuJoTQ

TRiD

Unknown!

Exiftool

Field	Value
error	exit status 1

Yara

No Matches
ShadowServer
Not found

time="2017-02-07T13:48:14Z" level=fatal msg="Please supply a valid SHA1 hash to query NSRL with."

Floss

Decoded Strings

No Strings

Stack Strings
No Strings

F-Secure

Infected Result Engine Updated

true PS-MPC.0873.AD.Gen 11.10 build 68 20170130

time="2017-02-07T13:48:53Z" level=fatal msg="Command /etc/init.d/avgd timed out." category=av path="/malware/b9bfb323d15ad4669781cb93e3c8f01fd2ad37b60d77c43fbe57b0942fbc0598" plugin=avg

Bitdefender

Infected Result Engine Updated

true PS-MPC.0873.AD.Gen 7.90123 20170129

Sophos

Infected Result Engine Updated

true Lady 5.31.0 20170130

ClamAV

Infected Result Engine Updated

true Win.Trojan.RedArc-5 0.99.2 20170130

Infected	Result	Engine	Updated
true	PS-MPC.0873.AD.Gen	11.10 build 68	20170130

Infected	Result	Engine	Updated
true	PS-MPC.0873.AD.Gen	7.90123	20170129

Infected	Result	Engine	Updated
true	Lady	5.31.0	20170130

Infected	Result	Engine	Updated
true	Win.Trojan.RedArc-5	0.99.2	20170130

Describe the results you expected: I would like to have at least avast and/or avg working =)

Additional information you deem important (e.g. issue happens only occasionally): Issue happens every time (Exiftools seems to work from time to time) Thank you =)

blacktop commented 7 years ago

I think your plugins are out of date can you please run:

$ malice plugin update --all

blacktop commented 7 years ago

Actually, hmm, it looks like AVG is timing out. So all malice plugins run independently as well, so you can try:

$ docker run --rm -v `pwd`:/malware malice/avg --timeout 300 --table evil.malware

Which will run the AVG plugin by itself and increase the default timeout to 300 secs.

blacktop commented 7 years ago

It looks like however it is taking just the avg daemon more that 30secs to just start which I have never seen on my test machine... I need to update that plugin so that the CLI arg for timeout also affects the starting of the daemon, but that is strange that it would take so long.

It looks like you have a LOT of stopped containers too, you might want to run:

$ docker rm `docker ps --no-trunc --filter "status=exited" -q`

to clear them out.

You might want to go into `~/.malice/plugins.toml and disable the VT and totalhash plugins as you need API keys for them to work.

Can also try running the avast plugin by itself like this:

$ docker run --rm -v /path/to/malware:/malware:ro malice/avast -V --table FILE

And show me the output it should have DEBUG output (because of the -V flag is set) and show me why there is an index out of range error happening.

Thank you!

blacktop commented 7 years ago

The reason that I think exiftool is failing is that actually libmagic is failing because it is taking a long time to finish and there is a race condition happening. Either that is a pretty cool piece of malware or your machine is very slow?

UPDATE: your machine should be fine from seeing the specs up above, so I think that malware is prety cool 😎

blacktop commented 7 years ago

This is all great to know however, as I usually test with the same 4 pieces of malware and know that there are edge cases I haven't seen yet. Thank you for letting me know 👍

blacktop commented 7 years ago

are those two scans? I don 't understand why there is two NSRL and shadow server outputs? As well as a failed libmagic and a successful one?

fbarre96 commented 7 years ago

So !

The malware (downloaded here http://openmalware.org/search.cgi?search=Lady) MD5: 65bb7a968098bb6b3d62e7edf7cdae39 SHA1: 012d4de9f1439348d89dae0e3a2d1ddaf33f31ac SHA256: b9bfb323d15ad4669781cb93e3c8f01fd2ad37b60d77c43fbe57b0942fbc0598 OCID: 321376031 Original Filename: Virus.DOS.Lady.873 Added: 2007-01-15 16:35:31.175765
I deleted all exited dockers (thx for the command)
All plugins were up to date
I ran avg and avast standalone with 300 timeout, and it worked fine for both (except that avast say it is not infected but that is another story..)
It was only one scan.
I ran a full scan again with docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v pwd:/malice/samples malice/engine scan malware.exe I get the same output as before except for avg shows result. (avast still bugging ='( )

I think I'll just write a little script to run this two plugins apart from the others.

blacktop commented 7 years ago

@fbarre96 are you saying that Avast was still failing? If so can you post the output with the -V flag set?

fbarre96 commented 7 years ago

File

Field	Value
Name	malice
Path	/usr/local/bin/malice
Size	16.85 MB
MD5	d198cccd47ffd98c99ca61a97e9d7328
SHA1	98e32660543eb0a084956e3aea52a538b295c4b9
SHA256	5232dbfbba82e68b5c09a7b943dc4aad0ab7baae50a2eac10cac57dcdfe9731e

NSRL Database

Not Found :grey_question:
ShadowServer
Not found

panic: runtime error: index out of range

goroutine 1 [running]: panic(0x85cde0, 0xc420014090) /usr/local/go/src/runtime/panic.go:500 +0x1a1 main.ParseAvastOutput(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/maliceio/malice-avast/scan.go:119 +0x575 main.AvScan(0x78, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/maliceio/malice-avast/scan.go:84 +0x2b9 main.main.func3(0xc420082780, 0x0, 0x0) /go/src/github.com/maliceio/malice-avast/scan.go:318 +0x17f github.com/urfave/cli.HandleAction(0x83fe00, 0xc4200154a0, 0xc420082780, 0x0, 0x0) /go/src/github.com/urfave/cli/app.go:485 +0xd4 github.com/urfave/cli.(*App).Run(0xc42007cb60, 0xc42000c330, 0x3, 0x3, 0x0, 0x0) /go/src/github.com/urfave/cli/app.go:259 +0x74f main.main() /go/src/github.com/maliceio/malice-avast/scan.go:357 +0x78f

Magic

Field	Value
Mime	application/x-executable
Description	ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

SSDeep

98304:wc8KFQPNyXsulgYpbX7KvKi8KIm95+UKNun+is:wc8KFQPNyXsuOYpbLKvKHKIm9MZk+is

TRiD

signal: killed

Exiftool

Field	Value
error	context deadline exceeded

F-PROT

Infected	Result	Engine	Updated
false		4.6.5.141	20170129

time="2017-02-10T09:41:24Z" level=fatal msg="Command /etc/init.d/avgd timed out." category=av path="/malware/5232dbfbba82e68b5c09a7b943dc4aad0ab7baae50a2eac10cac57dcdfe9731e" plugin=avg

Yara

Rule	Description	Offset	Data
embedded_win_api	A non-Windows executable contains win32 API functions names	7793322	WriteFile
maldoc_suspicious_strings		7793162	ReadFile
PEiD_00055_Alias_PIX_Vivid_IMG_Graphicsformat	[Alias PIX/Vivid IMG Graphics format]	5811753	�� 0 P
PEiD_02402_UPolyX_V01____Delikon	[UPolyX V0.1 -> Delikon]	5154234	��

Comodo

Infected	Result	Engine	Updated
false		1.1	20170129

time="2017-02-10T09:41:51Z" level=fatal msg="Command /opt/f-secure/fsav/bin/fsav timed out." category=av path="/malware/5232dbfbba82e68b5c09a7b943dc4aad0ab7baae50a2eac10cac57dcdfe9731e" plugin=fsecure

Bitdefender

Infected	Result	Engine	Updated
false

ClamAV

Infected	Result	Engine	Updated
false

time="2017-02-10T09:42:01Z" level=fatal msg="Command savscan timed out." category=av path="/malware/5232dbfbba82e68b5c09a7b943dc4aad0ab7baae50a2eac10cac57dcdfe9731e" plugin=sophos

Here the scan I did today

blacktop commented 7 years ago

I have added the ability for each malice plugin to respect the MALICE_TIMEOUT env var. So you should be able to export MALICE_TIMEOUT=600 and have all plugins now have a 10min timeout in place.

blacktop commented 7 years ago

you might have to reinstall malice and make sure you remove the old ~/.malice folder or at least the ~/.malice/plugin.toml file so the new one will have the env flags.

I am going to work on the ability to upgrade configs automatically soon.

maliceio / malice

Plugins failure #44

File

NSRL Database

ShadowServer

Comodo

F-PROT

Magic

SSDeep

TRiD

Exiftool

Yara

ShadowServer

Floss

Decoded Strings

Stack Strings

F-Secure

Bitdefender

Sophos

ClamAV

File

NSRL Database

ShadowServer

Magic

SSDeep

TRiD

Exiftool

F-PROT

Yara

Comodo

Bitdefender

ClamAV