search

maliceio / malice

VirusTotal Wanna Be - Now with 100% more Hipster
Apache License 2.0
1.65k stars 265 forks source link

Elasticsearch and scan issue #72

Closed n0tis closed 5 years ago

n0tis commented 6 years ago

Output of go version:

(paste your output here)

Output of docker version:

Docker version 18.06.1-ce, build e68fc7a

Output of docker info:

Containers: 26
 Running: 1
 Paused: 0
 Stopped: 25
Images: 28
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-34-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 7.767GiB
Name: malice
ID: DHKD:RHYS:MA47:GFIC:AFR6:XCZZ:R7QO:V2VC:NIC7:SEN3:RG3D:MIHU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, Docker For Mac, Docker Toolbox, docker-machine, etc.): Ubuntu 18.04 VM in esxi, docker-ce

Steps to reproduce the issue:

  1. Installed malice with the Linux guide (https://github.com/maliceio/malice/blob/master/docs/installation/linux/install.md)
  2. Ran elasticsearch with sudo docker -D run -d -p 9200:9200 --name malice-elastic -e ES_JAVA_OPTS="-Xms2g -Xmx2g" malice/elasticsearch
  3. Trying to scan a file with two different scan commands:

sudo docker -D run --rm -v /var/run/docker.sock:/var/run/docker.sock -v pwd:/malice/samples -e MALICE_VT_API=$MALICE_VT_API malice/engine scan test.txt

sudo malice -D scan test.txt

Describe the results you received:

Running the command initially to start elasticsearch only starts elasticsearch container. I have to run that same command again and then I see the KIbana container start. Having both containers running, I can successfully connect to elasticsearch and kibana via a browser which confirms they are both listening. Now when I try to scan a file with (sudo docker -D run --rm -v /var/run/docker.sock:/var/run/docker.sock -v pwd:/malice/samples -e MALICE_VT_API=$MALICE_VT_API malice/engine scan test.txt), I get the error message

"FATA[0000] scan cmd failed to store file info: failed to connect to database: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp 127.0.0.1:9200: connect: connection refused DEBU[0001] [hijack] End of stdout "

When I attempt with the next command (sudo malice -D scan test.txt):

sudo malice -D scan test.txt DEBU[0000] Malice config loaded from: /home/spyd3r/.malice/config/config.toml DEBU[0000] config.toml version: v0.3.11, malice version: 0.3.11 DEBU[0000] Malice plugins loaded from: /home/spyd3r/.malice/plugins/plugins.toml DEBU[0000] Using 1 PROCS
DEBU[0000] Malice Version: 0.3.11, commit 9e7f91e08345f227b4ddfe3690bc5ee52d1c6feb, built at 2017-07-22T23:52:36Z DEBU[0000] Running inside Docker...
DEBU[0000] Connected to docker daemon client ip=localhost port=2375 DEBU[0000] Searching for container: malice-elastic env=development DEBU[0000] name: malice-elastic container.Name: malice-kibana DEBU[0000] MATCH: false
DEBU[0000] name: malice-elastic container.Name: malice-elastic DEBU[0000] MATCH: true
DEBU[0000] Container FOUND: malice-elastic env=development DEBU[0000] Elasticsearch is running. image="malice/elasticsearch:6.4" ip=172.17.0.3 network=default DEBU[0000] Attempting to PING to: http://localhost:9200 DEBU[0000] ElasticSearch connection successful. address="http://localhost:9200" cluster=elasticsearch code=200 version=6.4.0 2018/10/01 14:48:38 elastic: Error 406 (Not Acceptable)

Describe the results you expected: Successful processing of a file and those results reflecting in Elastic\Kibana.

Additional information you deem important (e.g. issue happens only occasionally): Looking up the" elastic:Error 406" error message shows it could be related to stricter elasticsearch content-type header requirement in version 6+. Just want to check if somebody else has experienced behavior, and also if there is a known workaround\fix.

Thanks

draske79 commented 5 years ago

Running 18.04 in VM as well. I do get the same error

"FATA[0000] scan cmd failed to store file info: failed to connect to database: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp 127.0.0.1:9200: connect: connection refused

On a normal scan : malice scan sample.sh

I increased the RAM in my VM to 4 GB, removed all docker volumes and reinstalled malice. The problem occurred once but on the second scan everything went fine. If I have more info I will let you know

draske79 commented 5 years ago

I made some more tests. My feeling is that malice does not wait long enough for elasticsearch to come online. It fails with error above. Often running the same scan again a little later works. Probably in the meantime elasticsearch came online

blacktop commented 5 years ago

Can you try increasing the timeout in the config:

najashark commented 5 years ago

i also have the same problem 

ubuntu@ubuntu:~$ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v `pwd`:/malice/samples -e MALICE_VT_API=$MALICE_VT_API malice/engine --debug scan .bashrc
DEBU[0000] Malice config loaded from: /malice/config/config.toml
DEBU[0000] config.toml version: v0.3.24, malice version: dev
DEBU[0000] Malice plugins loaded from plugins/bindata.go
DEBU[0000] Using 2 PROCS
DEBU[0000] Malice Version: dev, commit none, built at unknown
DEBU[0000] Running inside Docker...
DEBU[0000] Connected to docker daemon client             ip=localhost port=2375
DEBU[0000] Searching for container: malice-elastic       env=development
DEBU[0000] name:  malice-elastic   container.Name:  gifted_hamilton
DEBU[0000] MATCH:  false
DEBU[0000] name:  malice-elastic   container.Name:  malice-kibana
DEBU[0000] MATCH:  false
DEBU[0000] name:  malice-elastic   container.Name:  malice-elastic
DEBU[0000] MATCH:  true
DEBU[0000] Container FOUND: malice-elastic               env=development
DEBU[0000] attempting to PING to: http://localhost:9200
DEBU[0000] Searching for image: malice/nsrl:sha1         env=development
DEBU[0000] Image FOUND: malice/nsrl:sha1                 env=development
DEBU[0000] Searching for image: malice/virustotal        env=development
DEBU[0000] Image FOUND: malice/virustotal                env=development
DEBU[0000] Searching for image: malice/shadow-server     env=development
DEBU[0000] Image FOUND: malice/shadow-server             env=development
DEBU[0000] Searching for image: malice/fileinfo          env=development
DEBU[0000] Image FOUND: malice/fileinfo                  env=development
DEBU[0000] Searching for image: malice/yara:neo23x0      env=development
DEBU[0000] Image FOUND: malice/yara:neo23x0              env=development
DEBU[0000] Searching for image: malice/avast             env=development
DEBU[0000] Image FOUND: malice/avast                     env=development
DEBU[0000] Searching for image: malice/avg               env=development
DEBU[0000] Image FOUND: malice/avg                       env=development
DEBU[0000] Searching for image: malice/bitdefender       env=development
DEBU[0000] Image FOUND: malice/bitdefender               env=development
DEBU[0000] Searching for image: malice/clamav            env=development
DEBU[0000] Image FOUND: malice/clamav                    env=development
DEBU[0000] Searching for image: malice/comodo            env=development
DEBU[0000] Image FOUND: malice/comodo                    env=development
DEBU[0000] Searching for image: quay.io/blacktop/drweb   env=development
DEBU[0000] Image FOUND: quay.io/blacktop/drweb           env=development
DEBU[0000] Searching for image: malice/escan             env=development
DEBU[0000] Image FOUND: malice/escan                     env=development
DEBU[0000] Searching for image: malice/fprot             env=development
DEBU[0000] Image FOUND: malice/fprot                     env=development
DEBU[0000] Searching for image: malice/fsecure           env=development
DEBU[0000] Image FOUND: malice/fsecure                   env=development
DEBU[0000] Searching for image: malice/mcafee            env=development
DEBU[0000] Image FOUND: malice/mcafee                    env=development
DEBU[0000] Searching for image: malice/sophos            env=development
DEBU[0000] Image FOUND: malice/sophos                    env=development
DEBU[0000] Searching for image: malice/windows-defender  env=development
DEBU[0000] Image FOUND: malice/windows-defender          env=development
DEBU[0000] Searching for image: malice/zoner             env=development
DEBU[0000] Image FOUND: malice/zoner                     env=development
DEBU[0000] Searching for image: malice/pescan            env=development
DEBU[0000] Image FOUND: malice/pescan                    env=development
DEBU[0000] Searching for image: malice/floss             env=development
DEBU[0000] Image FOUND: malice/floss                     env=development
DEBU[0000] Searching for image: malice/pdf               env=development
DEBU[0000] Image FOUND: malice/pdf                       env=development
DEBU[0000] All enabled plugins are installed.
#### File
| Field  | Value                                                            |
| ------ | ---------------------------------------------------------------- |
| Name   | .bashrc                                                          |
| Path   | .bashrc                                                          |
| Size   | 3.106kB                                                          |
| MD5    | cf277664b1771217d7006acdea006db1                                 |
| SHA1   | 17d380175c89fb145357edd7f1356f6274bfc762                         |
| SHA256 | 34fbc467b8c624d92abcdf3edcf35ee46032618a6f23b210efab0e6824978126 |
DEBU[0000] Searching for Network: malice                 env=development
DEBU[0000] Network FOUND: malice                         env=development
DEBU[0000] Searching for volume: malice                  env=development
DEBU[0000] Volume FOUND: malice                          env=development
DEBU[0000] Volume malice found.
DEBU[0000] Searching for container: copy2volume          env=development
DEBU[0000] name:  copy2volume   container.Name:  gifted_hamilton
DEBU[0000] MATCH:  false
DEBU[0000] name:  copy2volume   container.Name:  malice-kibana
DEBU[0000] MATCH:  false
DEBU[0000] name:  copy2volume   container.Name:  malice-elastic
DEBU[0000] MATCH:  false
DEBU[0000] Container NOT Found: copy2volume              env=development
DEBU[0000] Searching for image: busybox                  env=development
DEBU[0000] Image FOUND: busybox                          env=development
DEBU[0000] Image `busybox` already pulled.               env=development exisits=true
DEBU[0000] First statContainerPath call.                 SampledsDir=/malice/samples container.Name=/copy2volume dstInfo="{/malice/34fbc467b8c624d92abcdf3edcf35ee46032618a6f23b210efab0e6824978126 false false }" dstStat="{34fbc467b8c624d92abcdf3edcf35ee46032618a6f23b210efab0e6824978126 3106 -rw-r--r-- 2014-02-20 02:43:56 +0000 UTC }" file.Path=.bashrc volSavePath=/malice/34fbc467b8c624d92abcdf3edcf35ee46032618a6f23b210efab0e6824978126
DEBU[0000] Sample .bashrc already in malice volume.
DEBU[0000] Removing container: 12654b6c9147bd2b5e0b930341e17449d775ce42d5fbba2d7ff1c77ac293806c
DEBU[0000] attempting to PING to: http://localhost:9200
FATA[0000] scan cmd failed to store file info: failed to connect to database: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp 127.0.0.1:9200: connect: connection refused

if i curl the elasticsearch, intended output is printed

ubuntu@ubuntu:~$ curl http://localhost:9200/
{
  "name" : "Kf8utcU",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "xBq_MiiXQDibjVCymHpDzg",
  "version" : {
    "number" : "6.4.0",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "595516e",
    "build_date" : "2018-08-17T23:18:47.308994Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

I have already increaase the timeout to 200, but the error stay the same

blacktop commented 5 years ago

please see #80

please reopen if that is not the issue