search

maliceio / malice

VirusTotal Wanna Be - Now with 100% more Hipster
Apache License 2.0
1.63k stars 266 forks source link

scanning a file result in crash of elasticsearch and fatal error in malice #82

Closed changemenemo closed 5 years ago

changemenemo commented 5 years ago

Describe the bug

every plugin result in failed to initalize elasticsearch

To Reproduce simple pdf download on internet -> malice scan test.pdf

Expected behavior

result of scans displayed in the terminal Environment (please complete the following information):

Output of docker version:

Client: Docker Engine - Community
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:47:43 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:55:00 2018
  OS/Arch:          linux/amd64
  Experimental:     true

Output of docker info:

Containers: 4
 Running: 1
 Paused: 0
 Stopped: 3
Images: 49
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.125-linuxkit
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: linuxkit-025000000001
ID: BWXS:IEZB:NXVZ:5B4F:UCQH:OF3Z:P2HU:QYQD:EH6Z:3WPG:SPTA:U74X
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 39
 Goroutines: 64
 System Time: 2018-11-28T13:05:49.977260407Z
 EventsListeners: 2
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Additional environment details (AWS, VirtualBox, physical, Docker For Mac, Docker Toolbox, docker-machine, etc.): result of docker logs

[2018-11-28T13:03:37,969][WARN ][o.e.c.l.LogConfigurator  ] [unknown] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  /usr/share/elasticsearch/config/log4j2.properties
[2018-11-28T13:03:38,200][INFO ][o.e.e.NodeEnvironment    ] [E2F-SSQ] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [38gb], net total_space [58.4gb], types [ext4]
[2018-11-28T13:03:38,200][INFO ][o.e.e.NodeEnvironment    ] [E2F-SSQ] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-11-28T13:03:38,211][INFO ][o.e.n.Node               ] [E2F-SSQ] node name derived from node ID [E2F-SSQcSgip1JUxcYTW6g]; set [node.name] to override
[2018-11-28T13:03:38,211][INFO ][o.e.n.Node               ] [E2F-SSQ] version[6.5.0], pid[1], build[oss/tar/816e6f6/2018-11-09T18:58:36.352602Z], OS[Linux/4.9.125-linuxkit/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-11-28T13:03:38,211][INFO ][o.e.n.Node               ] [E2F-SSQ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/usr/share/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar]
[2018-11-28T13:03:38,902][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [aggs-matrix-stats]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [analysis-common]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [ingest-common]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [lang-expression]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [lang-mustache]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [lang-painless]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [mapper-extras]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [parent-join]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [percolator]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [rank-eval]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [reindex]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [repository-url]
[2018-11-28T13:03:38,903][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [transport-netty4]
[2018-11-28T13:03:38,904][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] loaded module [tribe]
[2018-11-28T13:03:38,904][INFO ][o.e.p.PluginsService     ] [E2F-SSQ] no plugins loaded
[2018-11-28T13:03:41,281][INFO ][o.e.d.DiscoveryModule    ] [E2F-SSQ] using discovery type [zen] and host providers [settings]
[2018-11-28T13:03:41,715][INFO ][o.e.n.Node               ] [E2F-SSQ] initialized
[2018-11-28T13:03:41,715][INFO ][o.e.n.Node               ] [E2F-SSQ] starting ...
[2018-11-28T13:03:41,877][INFO ][o.e.t.TransportService   ] [E2F-SSQ] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}
[2018-11-28T13:03:41,894][INFO ][o.e.b.BootstrapChecks    ] [E2F-SSQ] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-11-28T13:03:44,935][INFO ][o.e.c.s.MasterService    ] [E2F-SSQ] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {E2F-SSQ}{E2F-SSQcSgip1JUxcYTW6g}{jxd5QjpPS4i15l1JbiuSjw}{172.17.0.2}{172.17.0.2:9300}
[2018-11-28T13:03:44,938][INFO ][o.e.c.s.ClusterApplierService] [E2F-SSQ] new_master {E2F-SSQ}{E2F-SSQcSgip1JUxcYTW6g}{jxd5QjpPS4i15l1JbiuSjw}{172.17.0.2}{172.17.0.2:9300}, reason: apply cluster state (from master [master {E2F-SSQ}{E2F-SSQcSgip1JUxcYTW6g}{jxd5QjpPS4i15l1JbiuSjw}{172.17.0.2}{172.17.0.2:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-11-28T13:03:44,978][INFO ][o.e.h.n.Netty4HttpServerTransport] [E2F-SSQ] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2018-11-28T13:03:44,978][INFO ][o.e.n.Node               ] [E2F-SSQ] started
[2018-11-28T13:03:45,230][INFO ][o.e.g.GatewayService     ] [E2F-SSQ] recovered [3] indices into cluster_state
[2018-11-28T13:03:45,507][WARN ][r.suppressed             ] [E2F-SSQ] path: /malice/samples/IapfWmcBxXBSfaduVwcC, params: {index=malice, id=IapfWmcBxXBSfaduVwcC, type=samples}
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [malice][samples][IapfWmcBxXBSfaduVwcC]: routing [null]]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:224) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:211) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1200(TransportSingleShardAction.java:140) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:263) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1130) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1247) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1221) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.onFailure(TransportSingleShardAction.java:107) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:708) [elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-6.5.0.jar:6.5.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.elasticsearch.transport.RemoteTransportException: [E2F-SSQ][172.17.0.2:9300][indices:data/read/get[s]]
Caused by: org.elasticsearch.index.shard.IllegalIndexShardStateException: CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED]
    at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1511) ~[elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:891) ~[elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:165) ~[elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:87) ~[elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:79) ~[elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:87) ~[elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:43) ~[elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.doRun(TransportSingleShardAction.java:112) ~[elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) ~[elasticsearch-6.5.0.jar:6.5.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.0.jar:6.5.0]
    ... 3 more
[2018-11-28T13:03:46,056][INFO ][o.e.c.r.a.AllocationService] [E2F-SSQ] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[malice][0]] ...]).
[2018-11-28T13:03:46,695][INFO ][o.e.c.m.MetaDataMappingService] [E2F-SSQ] [malice/ayNdUZsZQkedL6EtHbDDZQ] update_mapping [samples]

Additional context Add any other context about the problem here.

blacktop commented 5 years ago

Did you try these? https://github.com/maliceio/malice/blob/master/docs/KnownBugs.md

changemenemo commented 5 years ago

Since it was on Macos (I know Linux base), I wasn't sure and I wanted your views first. I'm going to test it.

De : blacktop notifications@github.com Envoyé : mercredi, novembre 28, 2018 2:13 PM À : maliceio/malice Cc : Impostor syndrom; Author Objet : Re: [maliceio/malice] scanning a file result in crash of elasticsearch and fatal error in malice (#82)

Did you try these? https://github.com/maliceio/malice/blob/master/docs/KnownBugs.md

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/maliceio/malice/issues/82#issuecomment-442441970, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJFPg0tWUYcGQDr851aB8eer5YCW-s9Bks5uzovsgaJpZM4Y3kRd.

changemenemo commented 5 years ago

okey by updating the VM to 4G, I got some results. and it seems that the process can go through the whole protocol without crashing.

I have still a lot of failed to update sample in a lot of plugin. Should I do a bug report for that or is it normal? failed: Yara, mcafee, fprot, escan, clamav, comodo, fsecure

succeeded: zoner, avast, dr web, bitdefender, sophos.

It stays stuck after the result of sophos but the container didn't crash... but the vcpus allocated are working for sure.

it finally ended after 11 mins with the pdf parser results and a long list of number.

changemenemo commented 5 years ago

6G of RAM for a 30MB pdf. Don't know if the size of the file has anything to do with it but so you know. still a 409 error with virustotal time="2018-11-29T00:45:02Z" level=fatal msg="failed to index malice/virustotal results: failed to update sample with id: RYrtXGcBjIpm6xqHErD5: elastic: Error 409 (Conflict): [samples][RYrtXGcBjIpm6xqHErD5]: version conflict, current version [2] is different than the one provided [1] [type=version_conflict_engine_exception]" category=intel hash= plugin=virustotal