malice/elasticsearch:6.3 will not stay alive.

spennymac commented 6 years ago

Output of go version:

go version go1.10.3 darwin/amd64

Output of docker version:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:13:02 2018
 OS/Arch:      darwin/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:22:38 2018
  OS/Arch:      linux/amd64
  Experimental: true

Output of docker info:

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 158
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.87-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 992.6MiB
Name: linuxkit-025000000001
ID: AXE2:EZW6:46H6:U5W4:YJFK:QD53:UHDA:5YRY:LXS4:WEEU:FZ37:H4J7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 23
 Goroutines: 44
 System Time: 2018-08-21T15:43:41.518811369Z
 EventsListeners: 3
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Contents of plugins.toml

[[plugin]]                                                                                                                                                                                                                                                                    
  enabled = true                                                                   
  name = "virustotal"                                                              
  description = "VirusTotal - files scan and hash lookup"                          
  category = "intel"                                                               
  image = "malice/virustotal"                                                      
  repository = "https://github.com/malice-plugins/virustotal.git"                  
  build = false                                                                    
  upload = false # Set upload to `true` if you want to upload sample to virustotal.com
  apikey = "2b5175c8501015152e47c5fa3950d41bc0444218f3e7490159d4f7c776d2c41e" # This is a test key, please replace with your own 
  cmd = "lookup"                                                                   
  mime = "hash"                                                                    
  hashtypes = [ "md5", "sha1", "sha256" ]                                          
  env = ["MALICE_VT_API", "MALICE_TIMEOUT"]

What am I doing

I built from source and trying to run a scan. Elasticsearch does not stay alive. At first it was bootstrap checks , so then i was able to get it to run via this command:

docker run -e discovery.type=single-node  --name malice-elastic malice/elasticsearch:6.3

But now running into virus total container not being started.

Am I missing something?

blacktop commented 6 years ago

So I was also experiencing that issue a while ago and the logs from elasticsearch: docker logs malice-elasticsearch we saying that elasticsearch was making the malice index read only because of a storage high water mark. So it thought my hard drive was going to fill up so it stopped indexing. Which is VERY weird because my laptop has 400GB free (so maybe the malware infecting my laptop was exfiling a lot of my personal data in rars and thats why??) 😖 😆

Anyway, I just tried running it now and it was unstable and a few plugins were failing, but I let the database sit for a few minutes 15mins and now it is VERY stable and is having no problems???

I might work on adding support for other databases this weekend so people don't have to mess around with the delicate finicky elasticsearch. At least not right out of the gates.

blacktop commented 6 years ago

I upgraded to elasticsearch:6.4 not sure if that will fix for you. I am not longer seeing the "high water mark" I also upgraded Docker for Mac so maybe that helped as well??

FYI, I am adding the ability to use external elasticsearch to malice so you don't have to use my elasticsearch docker images etc

blacktop commented 5 years ago

I updated to elasticsearch 6.5.0 in the latest version and also created this as a ref #80 I'll make it into a FAQ today as well

maliceio / malice

malice/elasticsearch:6.3 will not stay alive. #70