search

maliceio / malice

VirusTotal Wanna Be - Now with 100% more Hipster
Apache License 2.0
1.65k stars 265 forks source link

Malice issue ubuntu 16.04 #79

Closed rufftruffles closed 5 years ago

rufftruffles commented 5 years ago

malice@malice:~$ go version go version go1.11.2 linux/amd64 malice@malice:~$ malice scan eicar.pdf ERRO[0000] database is NOT running, starting now...
panic: runtime error: index out of range

goroutine 1 [running]: github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.readTermInfo(0xc000020960, 0x1e, 0x0, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:232 +0xcdf github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.OpenTermInfo(0xc000020065, 0xe, 0xc000020065, 0xe, 0xc000379790) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:45 +0x287 github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage.DisplayJSONMessagesStream(0xbbc740, 0xc00005e580, 0xbbc940, 0xc000092008, 0x1, 0x1, 0x0, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage/jsonmessage.go:225 +0x55c github.com/maliceio/malice/malice/docker/client/image.Pull(0xc00001c720, 0xc00028b660, 0x18, 0xb0ef8b, 0x6) /Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/image/image.go:43 +0x181 github.com/maliceio/malice/malice/docker/client/container.checkContainerRequirements(0xc00001c720, 0xc00028b620, 0xe, 0xc00028b660, 0x18, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/container/utils.go:189 +0x92a github.com/maliceio/malice/malice/docker/client/container.Start(0xc00001c720, 0x0, 0x0, 0x0, 0xc00028b620, 0xe, 0xc00028b660, 0x18, 0x0, 0xc0003d8040, ...) /Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/container/start.go:36 +0xf9 github.com/maliceio/malice/malice/database.Start(0xc00001c720, 0x0, 0x0, 0x0, 0x0, 0xc00028b6a0, 0x15, 0x0, 0x0, 0x0, ...) /Users/blacktop/go/src/github.com/maliceio/malice/malice/database/database.go:45 +0x273 github.com/maliceio/malice/commands.cmdScan(0x7ffc360d46ce, 0x9, 0x0, 0xc00022b600, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/commands/scan.go:62 +0xe26 github.com/maliceio/malice/commands.glob..func1(0xc0000aac60, 0x0, 0xc0000aac60) /Users/blacktop/go/src/github.com/maliceio/malice/commands/commands.go:25 +0x88 github.com/maliceio/malice/vendor/github.com/urfave/cli.HandleAction(0xa10320, 0xb37b88, 0xc0000aac60, 0xc00022b600, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:490 +0xc8 github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.Run(0xb0e2b0, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb1188e, 0xb, 0x0, ...) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:210 +0x990 github.com/maliceio/malice/vendor/github.com/urfave/cli.(*App).Run(0xc0000a5380, 0xc000086060, 0x3, 0x3, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:255 +0x687 main.main() /Users/blacktop/go/src/github.com/maliceio/malice/main.go:88 +0x4b2

I tried with multiple GO versions but nothing, any ideas?

Regards,

blacktop commented 5 years ago

does the same thing happen if you use the deb package? https://github.com/maliceio/malice/releases/tag/v0.3.24

rufftruffles commented 5 years ago

Yep, just installed it, below is the log:

malice@malice:~$ malice -D scan eicar.pdf DEBU[0000] Malice config loaded from: /home/malice/.malice/config/config.toml DEBU[0000] config.toml version: v0.3.24, malice version: 0.3.24 DEBU[0000] Malice plugins loaded from: /home/malice/.malice/plugins/plugins.toml DEBU[0000] Using 4 PROCS
DEBU[0000] Malice Version: 0.3.24, commit 4a800572a0d5f95db62dcb7bbee55985ba4ba629, built at 2018-09-09T20:28:58Z DEBU[0000] Running inside Docker...
DEBU[0000] Connected to docker daemon client ip=localhost port=2375 DEBU[0000] Searching for container: malice-elastic env=development DEBU[0000] name: malice-elastic container.Name: kibana DEBU[0000] MATCH: false
DEBU[0000] name: malice-elastic container.Name: elastic DEBU[0000] MATCH: false
DEBU[0000] Container NOT Found: malice-elastic env=development ERRO[0000] database is NOT running, starting now...
DEBU[0000] Searching for Network: malice env=development DEBU[0000] Network FOUND: malice env=development DEBU[0000] Searching for volume: malice env=development DEBU[0000] Volume FOUND: malice env=development DEBU[0000] Volume malice found.
DEBU[0000] Searching for container: malice-elastic env=development DEBU[0000] name: malice-elastic container.Name: kibana DEBU[0000] MATCH: false
DEBU[0000] name: malice-elastic container.Name: elastic DEBU[0000] MATCH: false
DEBU[0000] Container NOT Found: malice-elastic env=development DEBU[0000] Searching for image: malice/elasticsearch:6.4 env=development DEBU[0000] Image NOT Found: malice/elasticsearch:6.4 env=development DEBU[0000] Pulling Image malice/elasticsearch:6.4 env=development exisits=false panic: runtime error: index out of range

goroutine 1 [running]: github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.readTermInfo(0xc0003a12e0, 0x1e, 0x0, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:232 +0xcdf github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.OpenTermInfo(0xc000020065, 0xe, 0xc000020065, 0xe, 0xc000054a30) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:45 +0x287 github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage.DisplayJSONMessagesStream(0xbbc740, 0xc000396b40, 0xbbc940, 0xc000090008, 0x1, 0xc0001ef501, 0x0, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage/jsonmessage.go:225 +0x55c github.com/maliceio/malice/malice/docker/client/image.Pull(0xc0003aa720, 0xc000289660, 0x18, 0xb0ef8b, 0x6) /Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/image/image.go:43 +0x181 github.com/maliceio/malice/malice/docker/client/container.checkContainerRequirements(0xc0003aa720, 0xc000289620, 0xe, 0xc000289660, 0x18, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/container/utils.go:189 +0x92a github.com/maliceio/malice/malice/docker/client/container.Start(0xc0003aa720, 0x0, 0x0, 0x0, 0xc000289620, 0xe, 0xc000289660, 0x18, 0x0, 0xc00039a780, ...) /Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/container/start.go:36 +0xf9 github.com/maliceio/malice/malice/database.Start(0xc0003aa720, 0x0, 0x0, 0x0, 0x0, 0xc0002896a0, 0x15, 0x0, 0x0, 0x0, ...) /Users/blacktop/go/src/github.com/maliceio/malice/malice/database/database.go:45 +0x273 github.com/maliceio/malice/commands.cmdScan(0x7ffcc22376ce, 0x9, 0x0, 0xc000229600, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/commands/scan.go:62 +0xe26 github.com/maliceio/malice/commands.glob..func1(0xc0000a6c60, 0x0, 0xc0000a6c60) /Users/blacktop/go/src/github.com/maliceio/malice/commands/commands.go:25 +0x88 github.com/maliceio/malice/vendor/github.com/urfave/cli.HandleAction(0xa10320, 0xb37b88, 0xc0000a6c60, 0xc000229600, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:490 +0xc8 github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.Run(0xb0e2b0, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb1188e, 0xb, 0x0, ...) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:210 +0x990 github.com/maliceio/malice/vendor/github.com/urfave/cli.(*App).Run(0xc0000a3380, 0xc000094000, 0x4, 0x4, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:255 +0x687 main.main() /Users/blacktop/go/src/github.com/maliceio/malice/main.go:88 +0x4b2

rufftruffles commented 5 years ago

docker :

malice@malice:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eccf14ba5fd0 malice/kibana "/entrypoint.sh kiban" 16 minutes ago Up 16 minutes 0.0.0.0:5601->5601/tcp kibana 9bf7175ae809 malice/elasticsearch "/elastic-entrypoint." 27 minutes ago Up 27 minutes 0.0.0.0:9200->9200/tcp, 9300/tcp elastic

blacktop commented 5 years ago

other people have said the first scan fails because it also tries to create/start the database, but that subsequent scans do work, because the db is already running?

rufftruffles commented 5 years ago

When trying to install/update plugins:

malice@malice:~$ malice -D plugin update --all DEBU[0000] Malice config loaded from: /home/malice/.malice/config/config.toml DEBU[0000] config.toml version: v0.3.24, malice version: 0.3.24 DEBU[0000] Malice plugins loaded from: /home/malice/.malice/plugins/plugins.toml DEBU[0000] Using 4 PROCS
DEBU[0000] Malice Version: 0.3.24, commit 4a800572a0d5f95db62dcb7bbee55985ba4ba629, built at 2018-09-09T20:28:58Z DEBU[0000] Running inside Docker...
DEBU[0000] Connected to docker daemon client ip=localhost port=2375 panic: runtime error: index out of range

goroutine 1 [running]: github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.readTermInfo(0xc000020300, 0x1e, 0x0, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:232 +0xcdf github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.OpenTermInfo(0xc000020065, 0xe, 0xc000020065, 0xe, 0xc0003ce1e0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:45 +0x287 github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage.DisplayJSONMessagesStream(0xbbc740, 0xc00005e140, 0xbbc940, 0xc000092008, 0x1, 0x1, 0x0, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage/jsonmessage.go:225 +0x55c github.com/maliceio/malice/malice/docker/client/image.Pull(0xc0003a8720, 0xb0f593, 0x7, 0xb0ef8b, 0x6) /Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/image/image.go:43 +0x181 github.com/maliceio/malice/plugins.UpdateEnabledPlugins(0xc0003a8720) /Users/blacktop/go/src/github.com/maliceio/malice/plugins/plugins.go:248 +0x8f github.com/maliceio/malice/commands.cmdUpdatePlugin(0x0, 0x0, 0x1, 0x0, 0xc00022b6e0) /Users/blacktop/go/src/github.com/maliceio/malice/commands/plugin.go:161 +0x20f github.com/maliceio/malice/commands.glob..func8(0xc0000aaf20, 0x0, 0xc0000aaf20) /Users/blacktop/go/src/github.com/maliceio/malice/commands/commands.go:138 +0xc1 github.com/maliceio/malice/vendor/github.com/urfave/cli.HandleAction(0xa10320, 0xb37bc0, 0xc0000aaf20, 0xc00022b600, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:490 +0xc8 github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.Run(0xb0f147, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb12fee, 0xd, 0x0, ...) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:210 +0x990 github.com/maliceio/malice/vendor/github.com/urfave/cli.(App).RunAsSubcommand(0xc0000a5520, 0xc0000aac60, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:379 +0x7ef github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.startApp(0xb0effd, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb1ed92, 0x1f, 0x0, ...) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:298 +0x808 github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.Run(0xb0effd, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb1ed92, 0x1f, 0x0, ...) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:98 +0x1237 github.com/maliceio/malice/vendor/github.com/urfave/cli.(App).Run(0xc0000a5380, 0xc000096000, 0x5, 0x5, 0x0, 0x0) /Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:255 +0x687 main.main() /Users/blacktop/go/src/github.com/maliceio/malice/main.go:88 +0x4b2

blacktop commented 5 years ago

I think it was a bad design decision on my part to have malware create the DBs 🤔

rufftruffles commented 5 years ago

I think it was a bad design decision on my part to have malware create the DBs

haha, what should I do in this scenario now?

blacktop commented 5 years ago

can you tell me more about your host? I ran this on an AWS node when I filled the demo.malice.io instance will all of its data

blacktop commented 5 years ago

I used debian package and it worked perfectly :(

rufftruffles commented 5 years ago

can you tell me more about your host? I ran this on an AWS node when I filled the demo.malice.io instance will all of its data

It's running ubuntu 16.04 on a KVM based vm, installed docker-engine, followed te following doc to install go & malice using source https://github.com/maliceio/malice/blob/master/docs/installation/linux/install.md

then also started elastic search container.

rufftruffles commented 5 years ago

I can give you access to the vm if you need to take a look and issue a fix for future releases?

blacktop commented 5 years ago

did you do this step:

https://github.com/maliceio/malice#known-issues-warning

sudo sysctl -w vm.max_map_count=262144
rufftruffles commented 5 years ago

did you do this step:

https://github.com/maliceio/malice#known-issues-warning

sudo sysctl -w vm.max_map_count=262144

Yes, as soon as I setup the vm lol

blacktop commented 5 years ago

Another thing you can do is get elasticsearch running outside of docker/malice, then you can point malice to it with the env var MALICE_ELASTICSEARCH_URL=<host>:<port>

rufftruffles commented 5 years ago

Another thing you can do is get elasticsearch running outside of docker/malice, then you can point malice to it with the env var MALICE_ELASTICSEARCH_URL=<host>:<port>

I don't think that's gonna work, should I send you the vm credentials via email to take a look in your free time? I'm sure this will help a lot of people out there struggling with the same issue.

blacktop commented 5 years ago

I'm going to spin up a vagrantbox for xenial and check real quick

rufftruffles commented 5 years ago

I'm going to spin up a vagrantbox for xenial and check real quick

Perfecto!

blacktop commented 5 years ago

Ok so I have a solution... but you are NOT going to like it! 😬

It's a friggin BUG in one of the docker src code dependancies! :rage4:

I was able to recreate your issue and the fix for me was:

$ TERM="" malice plugin update clamav
blacktop commented 5 years ago

It looks like docker-it-self had the SAME issue and they solved a while back and it fixed it by overriding the dep with another repo since the repo is dead. I copied their solution and cut another release. Can you please test and let me know.

That was a VERY embarrassing bug, thank you for pointing that out to me!

rufftruffles commented 5 years ago

It looks like docker-it-self had the SAME issue and they solved a while back and it fixed it by overriding the dep with another repo since the repo is dead. I copied their solution and cut another release. Can you please test and let me know.

That was a VERY embarrassing bug, thank you for pointing that out to me!

Hey there! Your fix worked wonders, everything is fixed except for the elasticsearch: here are a few outputs:

alice@malice:~$ malice scan f.pdf
#### File
| Field  | Value                                                            |
| ------ | ---------------------------------------------------------------- |
| Name   | f.pdf                                                            |
| Path   | f.pdf                                                            |
| Size   | 2.061kB                                                          |
| MD5    | 911dd1610034027a924387d42f56bdf0                                 |
| SHA1   | 6ce8d59428b6a646ac5eb440b540e8984ece5b08                         |
| SHA256 | 4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 |
FATA[0001] scan cmd failed to store file info: failed to connect to database: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp [::1]:9200: connect: connection refused

The container was created when I ran malice scan first time:

malice@malice:~$ malice scan eicar.pdf
ERRO[0000] database is NOT running, starting now...     
ERRO[0000] Network malice does not exist, creating now...  env=development exisits=false network=malice
INFO[0000] Created Network: malice                       env=development name=malice
INFO[0000] Created Volume: malice                        env=development
6.5: Pulling from malice/elasticsearch
4fe2ade4980c: Pull complete 
c9dbc0055e45: Pull complete 
d4511882860e: Pull complete 
2772c7b6d4e2: Pull complete 
589015d5f852: Pull complete 
e1dae11492e9: Pull complete 
9ecd75eb0b8e: Pull complete 
0f42f265a9ba: Pull complete 
Digest: sha256:0fdbffc5b93cb612bf4d64c93b8627a6438d293a3b0394e0f4054545f99500b8
Status: Downloaded newer image for malice/elasticsearch:6.5
INFO[0012] elasticsearch container started               assigned_ip=172.17.0.2 docker_ip=localhost name=/malice-elastic port="[9200]" runtime_env=development
FATA[0032] failed to start to database: connecting to elasticsearch timed out after 20 seconds: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp [::1]:9200: connect: connection refused
malice@malice:~$ docker start 9ae59e8f4012
9ae59e8f4012
malice@malice:~$ malice scan f.pdf
#### File
| Field  | Value                                                            |
| ------ | ---------------------------------------------------------------- |
| Name   | f.pdf                                                            |
| Path   | f.pdf                                                            |
| Size   | 2.061kB                                                          |
| MD5    | 911dd1610034027a924387d42f56bdf0                                 |
| SHA1   | 6ce8d59428b6a646ac5eb440b540e8984ece5b08                         |
| SHA256 | 4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 |
**FATA[0002] scan cmd failed to store file info: failed to connect to database: failed to ping elasticsearch: Get http://localhost:9200/: read tcp [::1]:46522->[::1]:9200: read: connection reset by peer**
blacktop commented 5 years ago

can you check the elastic logs. please see https://github.com/maliceio/malice/issues/80

rufftruffles commented 5 years ago

Next steps: removed elastic container and redeployed manually:

malice@malice:~$ docker rm 9ae59e8f4012
9ae59e8f4012
malice@malice:~$ docker run -d --name elastic -p 9200:9200 malice/elasticsearch
Unable to find image 'malice/elasticsearch:latest' locally
latest: Pulling from malice/elasticsearch
4fe2ade4980c: Already exists 
c9dbc0055e45: Already exists 
d4511882860e: Already exists 
2772c7b6d4e2: Already exists 
589015d5f852: Already exists 
e1dae11492e9: Already exists 
9ecd75eb0b8e: Already exists 
0f42f265a9ba: Already exists 
Digest: sha256:c7dbed8f3054499e2d11991cab4aef641ba5a63b38874e9372915473a5ef5252
Status: Downloaded newer image for malice/elasticsearch:latest
241c1addf6be974697d1c14096de073d36f122f2561c4c5100bc571da2d8af27
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   7 seconds ago       Up 5 seconds        0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   12 seconds ago      Up 10 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   13 seconds ago      Up 12 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   15 seconds ago      Up 13 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   15 seconds ago      Up 14 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                       PORTS               NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   38 seconds ago      Exited (78) 18 seconds ago                       elastic
blacktop commented 5 years ago

It might also be that malice isn't giving elasticsearch enough time to start on your machine

rufftruffles commented 5 years ago

can you check the elastic logs. please see #80

There you go:

malice@malice:~$ docker logs -f elastic
[2018-11-24T14:35:00,648][WARN ][o.e.c.l.LogConfigurator  ] [unknown] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  /usr/share/elasticsearch/config/log4j2.properties
[2018-11-24T14:35:01,262][INFO ][o.e.e.NodeEnvironment    ] [YiAkBn1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [15.1gb], net total_space [28.5gb], types [ext3]
[2018-11-24T14:35:01,263][INFO ][o.e.e.NodeEnvironment    ] [YiAkBn1] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-11-24T14:35:01,269][INFO ][o.e.n.Node               ] [YiAkBn1] node name derived from node ID [YiAkBn1LScK9-toK0v9DBw]; set [node.name] to override
[2018-11-24T14:35:01,270][INFO ][o.e.n.Node               ] [YiAkBn1] version[6.5.0], pid[1], build[oss/tar/816e6f6/2018-11-09T18:58:36.352602Z], OS[Linux/4.4.0-109-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-11-24T14:35:01,270][INFO ][o.e.n.Node               ] [YiAkBn1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/usr/share/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar]
[2018-11-24T14:35:02,703][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [aggs-matrix-stats]
[2018-11-24T14:35:02,703][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [analysis-common]
[2018-11-24T14:35:02,703][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [ingest-common]
[2018-11-24T14:35:02,703][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [lang-expression]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [lang-mustache]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [lang-painless]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [mapper-extras]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [parent-join]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [percolator]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [rank-eval]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [reindex]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [repository-url]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [transport-netty4]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [tribe]
[2018-11-24T14:35:02,706][INFO ][o.e.p.PluginsService     ] [YiAkBn1] no plugins loaded
[2018-11-24T14:35:08,150][INFO ][o.e.d.DiscoveryModule    ] [YiAkBn1] using discovery type [zen] and host providers [settings]
[2018-11-24T14:35:08,920][INFO ][o.e.n.Node               ] [YiAkBn1] initialized
[2018-11-24T14:35:08,921][INFO ][o.e.n.Node               ] [YiAkBn1] starting ...
[2018-11-24T14:35:09,168][INFO ][o.e.t.TransportService   ] [YiAkBn1] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
[2018-11-24T14:35:09,187][INFO ][o.e.b.BootstrapChecks    ] [YiAkBn1] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-11-24T14:35:09,202][INFO ][o.e.n.Node               ] [YiAkBn1] stopping ...
[2018-11-24T14:35:09,346][INFO ][o.e.n.Node               ] [YiAkBn1] stopped
[2018-11-24T14:35:09,347][INFO ][o.e.n.Node               ] [YiAkBn1] closing ...
[2018-11-24T14:35:09,370][INFO ][o.e.n.Node               ] [YiAkBn1] closed
blacktop commented 5 years ago

whoa I don't understand how you can have so many elastics running at the same time, docker should have complained that you already had something listening on port 9200 ??

blacktop commented 5 years ago

so that looks like it needs your to run

echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
sudo sysctl -w vm.max_map_count=262144

I assume if you DID run sudo sysctl -w vm.max_map_count=262144 then you rebooted the vm? because it doesn't presist unless you write it to /etc/sysctl.conf

rufftruffles commented 5 years ago

whoa I don't understand how you can have so many elastics running at the same time, docker should have complained that you already had something listening on port 9200 ??

Oh no, take a look again, I kept running ps -a to see when the container dies, check the result of last ps -a (container died after 38 secs of startup):

malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                       PORTS               NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   38 seconds ago      Exited (78) 18 seconds ago                       elastic
blacktop commented 5 years ago

ah ok, also I think malice expects the container to be called malice-elastic

rufftruffles commented 5 years ago

ah ok, also I think malice expects the container to be called malice-elastic

Oh crap! I had reinstalled the vm and forgot to update max map count :D

blacktop commented 5 years ago

So ya, when the docker logs -f malice-elastic says:

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

It means you need to run: sudo sysctl -w vm.max_map_count=262144

rufftruffles commented 5 years ago

Getting some errors with the scan at the end of the result, let me update you in a sec.

blacktop commented 5 years ago

I'm going to add a feature in malice where it will scan the logs for malice-elastic automatically to help diagnose these types of issues because MOST people have them.

I already have one check for lack of RAM here: https://github.com/maliceio/malice/blob/master/malice/database/database.go#L84

rufftruffles commented 5 years ago

elastic died during scan:

malice@malice:~$ malice scan f.pdf
#### File
| Field  | Value                                                            |
| ------ | ---------------------------------------------------------------- |
| Name   | f.pdf                                                            |
| Path   | f.pdf                                                            |
| Size   | 2.061kB                                                          |
| MD5    | 911dd1610034027a924387d42f56bdf0                                 |
| SHA1   | 6ce8d59428b6a646ac5eb440b540e8984ece5b08                         |
| SHA256 | 4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 |
#### ShadowServer
##### AntiVirus
 - FirstSeen: 7/08/2015 6:03PM
 - LastSeen: 4/15/2018 11:08PM

| Vendor          | Signature        |
|:----------------|:-----------------|
| AVG | VB2.JLL |
| AVG7 | Exploit.PDF |
| AhnLab | EICAR_Test_File |
| AntiVir | HEUR/PDF.Obfuscated |
| Authentium | EICAR_Test_File |
| Avast | virus!!! |
| Avira | Eicar-Test-Signature |
| BitDefender | Trojan.Script.103116 |
| Clam | Pdf.Dropper.Agent-6299277-0 |
| Comodo | UnclassifiedMalware |
| DrWeb | EICAR |
| F-Secure | Virus:W32/Eicar.C |
| FProt | EICAR_Test_File |
| FSecure | Virus:W32/Eicar.C |
| Fortinet | EICAR_TEST_FILE |
| G-Data | Trojan.Script.103116 |
| GData | EICAR_TEST_FILE |
| Ikarus | EICAR-Test-File |
| Kaspersky | EICAR-Test-File |
| McAfee | PDF-Exploit!911DD1610034 |
| MicroWorld | EICAR-Test-FileZP |
| Microsoft | Virus:DOS/EICAR_Test_File |
| Norman | pdf:doslegacy/EICAR_Test_file_not_a_virus! |
| Panda | EICAR-AV-TEST-FILE |
| QuickHeal | Eicar.Sig.A |
| Sophos | Sus/PDFJs-S |
| TrendMicro | Eicar_test_file |
| VBA32 | EICAR-Test-File |
| Vba32 | EICAR-Test-File |
| Vexira | EICAR_test_file |
| VirusBuster | EICAR_test_file |

time="2018-11-24T14:43:51Z" level=fatal msg="failed to index malice/nsrl results: failed to update sample with id: YBstRmcB8jjQ7OwGJXEM: elastic: Error 409 (Conflict): [samples][YBstRmcB8jjQ7OwGJXEM]: version conflict, current version [2] is different than the one provided [1] [type=version_conflict_engine_exception]" category=intel hash=6CE8D59428B6A646AC5EB440B540E8984ECE5B08 plugin=nsrl
#### VirusTotal
| Ratio      | Link         | API         | Scanned     |
|------------|--------------|-------------|-------------|
| 62% | [link](https://www.virustotal.com/file/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513/analysis/1542970827/) | Public | 2018-11-23 11:00:27 |

#### Magic
| Field       | Value                  |
|-------------|------------------------|
| Mime        | application/pdf        |
| Description | PDF document, version 1.1 |

#### SSDeep
 - `48:ull2naNCsOB2l8ggv1KToW7RO2MJXbRTd2l6vRzToMxZmdKor:uz2naNCsO68yToUYJXbRTd0qToMnoJr`

#### TRiD
- Warning: file seems to be plain text/ASCII
- TrID is best suited to analyze binary files!
- 100.0% (.PDF) Adobe Portable Document Format (5000/1)

#### Exiftool
| Field       | Value                |
|-------------|----------------------|
| ExifToolVersionNumber  | 11.11        |
| FileSize  | 2.0 kB        |
| FileType  | PDF        |
| FileTypeExtension  | pdf        |
| Linearized  | No        |
| MIMEType  | application/pdf        |
| PDFVersion  | 1.1        |
| PageCount  | 1        |

time="2018-11-24T14:44:03Z" level=error msg="avast license has expired"
time="2018-11-24T14:44:04Z" level=error msg="please get a new one here: https://www.avast.com/linux-server-antivirus"
#### Yara
time="2018-11-24T14:44:07Z" level=fatal msg="signal: illegal instruction (core dumped)" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=windows_defender
#### Zoner
| Infected      | Result      | Engine      | Updated      |
|:-------------:|:-----------:|:-----------:|:------------:|
| false |  | 3107100 | 20180928 |

### pdf
#### [PDFiD]
 - **PDF Header:** `%PDF-1.1`
 - **Total Entropy:** `4.984534`
 - **Entropy In Streams:** `4.412538`
 - **Entropy Out Streams:** `4.901738`
 - **Count %% EOF:** `1`
 - **Data After EOF:** `0`

| Keyword     | Count     |
|-------------|-----------|
| obj      | 10        |
| endobj      | 10        |
| stream      | 2        |
| endstream      | 2        |
| xref      | 1        |
| trailer      | 1        |
| startxref      | 1        |
| /Page      | 1        |
| /Encrypt      | 0        |
| /ObjStm      | 0        |
| /JS      | 1        |
| /JavaScript      | 1        |
| /AA      | 0        |
| /OpenAction      | 0        |
| /AcroForm      | 0        |
| /JBIG2Decode      | 0        |
| /RichMedia      | 0        |
| /Launch      | 0        |
| /EmbeddedFile      | 1        |
| /XFA      | 0        |
| /Colors > 2^24      | 0        |

##### Embedded File
> **Score:** `1000`

**Reasons:**
 - `/EmbeddedFile` flag(s) are hex encoded
##### Name Obfuscation
> **Score:** `1000`
**Reasons:**
 - hex encoded flag(s) detected
##### Triage
> **Score:** `100`

**Reasons:**
- `/JS`: indicating javascript is present in the file.
- `/JavaScript`: indicating javascript is present in the file.

##### Suspicious Properties
> **Score:** `50`

**Reasons:**
- Page count of 1

#### [pdf-parser]
##### Stats
- `Comment: 2`
- `XREF: 1`
- `Trailer: 1`
- `StartXref: 1`
- `Indirect object: 10`
- ` 1: 5`
- `/#45mbeddedFile 1: 10`
- `/Action 1: 8`
- `/Annot 1: 7`
- `/Catalog 1: 1`
- `/Filespec 1: 9`
- `/Font 1: 6`
- `/Outlines 1: 2`
- `/Page 1: 4`
- `/Pages 1: 3`

##### TAGS
**file_name:**
- `EICAR.txt`

##### Carved Content
**EmbeddedFile:**
s<<++<<            /Names [(EICAR.txt) 9 0 R]
**JS:**
javascript
(this.exportDataObject({ cName: "EICAR.txt", nLaunch: 2 })    ; )

time="2018-11-24T14:44:52Z" level=fatal msg="exit status 150" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=avg

time="2018-11-24T14:45:22Z" level=fatal msg="failed to initalize elasticsearch: failed to connect to database: failed to ping elasticsearch: Get http://elasticsearch:9200/: EOF" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=fprot
time="2018-11-24T14:45:22Z" level=fatal msg="failed to initalize elasticsearch: failed to connect to database: failed to ping elasticsearch: Get http://elasticsearch:9200/: EOF" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=avast
time="2018-11-24T14:45:50Z" level=fatal msg="failed to initalize elasticsearch: failed to connect to database: failed to ping elasticsearch: Get http://elasticsearch:9200/: dial tcp 172.17.0.2:9200: connect: no route to host" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=mcafee
time="2018-11-24T14:45:51Z" level=fatal msg="failed to initalize elasticsearch: failed to connect to database: failed to ping elasticsearch: Get http://elasticsearch:9200/: dial tcp 172.17.0.2:9200: connect: no route to host" category=av path= plugin=comodo
time="2018-11-24T14:46:03Z" level=fatal msg="signal: killed" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=bitdefender
time="2018-11-24T14:46:05Z" level=fatal msg="signal: killed" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=escan
time="2018-11-24T14:46:06Z" level=fatal msg="signal: killed" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=clamav
time="2018-11-24T14:46:20Z" level=fatal msg="context deadline exceeded" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=drweb
time="2018-11-24T14:46:22Z" level=fatal msg="failed to initalize elasticsearch: failed to connect to database: failed to ping elasticsearch: Get http://elasticsearch:9200/: dial tcp 172.17.0.2:9200: connect: no route to host" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=fsecure
time="2018-11-24T14:46:24Z" level=fatal msg="failed to initalize elasticsearch: failed to connect to database: failed to ping elasticsearch: Get http://elasticsearch:9200/: dial tcp 172.17.0.2:9200: connect: no route to host" category=av path=/malware/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 plugin=sophos

Container logs:

malice@malice:~$ docker logs -f malice-elastic
[2018-11-24T14:43:28,596][WARN ][o.e.c.l.LogConfigurator  ] [unknown] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  /usr/share/elasticsearch/config/log4j2.properties
[2018-11-24T14:43:29,031][INFO ][o.e.e.NodeEnvironment    ] [FVmN6Mn] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [15.1gb], net total_space [28.5gb], types [ext3]
[2018-11-24T14:43:29,031][INFO ][o.e.e.NodeEnvironment    ] [FVmN6Mn] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-11-24T14:43:29,033][INFO ][o.e.n.Node               ] [FVmN6Mn] node name derived from node ID [FVmN6MnFQgGQDesSO9AHuQ]; set [node.name] to override
[2018-11-24T14:43:29,033][INFO ][o.e.n.Node               ] [FVmN6Mn] version[6.5.0], pid[1], build[oss/tar/816e6f6/2018-11-09T18:58:36.352602Z], OS[Linux/4.4.0-109-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-11-24T14:43:29,034][INFO ][o.e.n.Node               ] [FVmN6Mn] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/usr/share/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar]
[2018-11-24T14:43:30,165][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [aggs-matrix-stats]
[2018-11-24T14:43:30,166][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [analysis-common]
[2018-11-24T14:43:30,166][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [ingest-common]
[2018-11-24T14:43:30,166][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [lang-expression]
[2018-11-24T14:43:30,166][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [lang-mustache]
[2018-11-24T14:43:30,166][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [lang-painless]
[2018-11-24T14:43:30,166][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [mapper-extras]
[2018-11-24T14:43:30,166][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [parent-join]
[2018-11-24T14:43:30,167][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [percolator]
[2018-11-24T14:43:30,167][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [rank-eval]
[2018-11-24T14:43:30,167][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [reindex]
[2018-11-24T14:43:30,167][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [repository-url]
[2018-11-24T14:43:30,167][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [transport-netty4]
[2018-11-24T14:43:30,167][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] loaded module [tribe]
[2018-11-24T14:43:30,168][INFO ][o.e.p.PluginsService     ] [FVmN6Mn] no plugins loaded
[2018-11-24T14:43:35,549][INFO ][o.e.d.DiscoveryModule    ] [FVmN6Mn] using discovery type [zen] and host providers [settings]
[2018-11-24T14:43:36,654][INFO ][o.e.n.Node               ] [FVmN6Mn] initialized
[2018-11-24T14:43:36,654][INFO ][o.e.n.Node               ] [FVmN6Mn] starting ...
[2018-11-24T14:43:36,911][INFO ][o.e.t.TransportService   ] [FVmN6Mn] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
[2018-11-24T14:43:36,932][INFO ][o.e.b.BootstrapChecks    ] [FVmN6Mn] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-11-24T14:43:40,013][INFO ][o.e.c.s.MasterService    ] [FVmN6Mn] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {FVmN6Mn}{FVmN6MnFQgGQDesSO9AHuQ}{b4ip9r6TRH2OeHPEZY9aAA}{172.17.0.2}{172.17.0.2:9300}
[2018-11-24T14:43:40,022][INFO ][o.e.c.s.ClusterApplierService] [FVmN6Mn] new_master {FVmN6Mn}{FVmN6MnFQgGQDesSO9AHuQ}{b4ip9r6TRH2OeHPEZY9aAA}{172.17.0.2}{172.17.0.2:9300}, reason: apply cluster state (from master [master {FVmN6Mn}{FVmN6MnFQgGQDesSO9AHuQ}{b4ip9r6TRH2OeHPEZY9aAA}{172.17.0.2}{172.17.0.2:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-11-24T14:43:40,079][INFO ][o.e.h.n.Netty4HttpServerTransport] [FVmN6Mn] publish_address {172.17.0.2:9200}, bound_addresses {[::]:9200}
[2018-11-24T14:43:40,079][INFO ][o.e.n.Node               ] [FVmN6Mn] started
[2018-11-24T14:43:40,125][INFO ][o.e.g.GatewayService     ] [FVmN6Mn] recovered [0] indices into cluster_state
[2018-11-24T14:43:40,790][INFO ][o.e.c.m.MetaDataCreateIndexService] [FVmN6Mn] [malice] creating index, cause [api], templates [], shards [1]/[0], mappings [samples]
[2018-11-24T14:43:41,425][INFO ][o.e.c.r.a.AllocationService] [FVmN6Mn] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[malice][0]] ...]).
[2018-11-24T14:43:50,780][INFO ][o.e.c.m.MetaDataMappingService] [FVmN6Mn] [malice/fH9R_-FETwW09kKAwnAseg] update_mapping [samples]
[2018-11-24T14:43:50,929][INFO ][o.e.c.m.MetaDataMappingService] [FVmN6Mn] [malice/fH9R_-FETwW09kKAwnAseg] update_mapping [samples]
[2018-11-24T14:43:58,315][INFO ][o.e.c.m.MetaDataMappingService] [FVmN6Mn] [malice/fH9R_-FETwW09kKAwnAseg] update_mapping [samples]
[2018-11-24T14:44:01,654][WARN ][o.e.m.j.JvmGcMonitorService] [FVmN6Mn] [gc][young][22][5] duration [2.9s], collections [1]/[3.6s], total [2.9s]/[3.1s], memory [369.7mb]->[198.8mb]/[990.7mb], all_pools {[young] [243.4mb]->[1.3mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [93mb]->[165.1mb]/[691.2mb]}
[2018-11-24T14:44:01,666][WARN ][o.e.m.j.JvmGcMonitorService] [FVmN6Mn] [gc][22] overhead, spent [2.9s] collecting in the last [3.6s]
[2018-11-24T14:44:06,863][INFO ][o.e.c.m.MetaDataMappingService] [FVmN6Mn] [malice/fH9R_-FETwW09kKAwnAseg] update_mapping [samples]
[2018-11-24T14:44:08,592][INFO ][o.e.c.m.MetaDataMappingService] [FVmN6Mn] [malice/fH9R_-FETwW09kKAwnAseg] update_mapping [samples]
[2018-11-24T14:44:14,347][INFO ][o.e.c.m.MetaDataMappingService] [FVmN6Mn] [malice/fH9R_-FETwW09kKAwnAseg] update_mapping [samples]
[2018-11-24T14:44:20,733][WARN ][o.e.m.j.JvmGcMonitorService] [FVmN6Mn] [gc][young][37][6] duration [4.6s], collections [1]/[5s], total [4.6s]/[7.7s], memory [431mb]->[197.1mb]/[990.7mb], all_pools {[young] [232.6mb]->[1.9mb]/[266.2mb]}{[survivor] [33.2mb]->[9mb]/[33.2mb]}{[old] [165.1mb]->[186.2mb]/[691.2mb]}
[2018-11-24T14:44:20,734][WARN ][o.e.m.j.JvmGcMonitorService] [FVmN6Mn] [gc][37] overhead, spent [4.6s] collecting in the last [5s]
blacktop commented 5 years ago

is that all the elasticsearch logs? it doesn't say that it died

blacktop commented 5 years ago

how much RAM does your VM have because ya it looks like elasticsearch is having trouble keeping up

rufftruffles commented 5 years ago

Made it work, increated system ram to 10 gig, was having issues at 8 gig thats weird. Seems like its working perfectly fine now. Finally, thanks to you!

blacktop commented 5 years ago

Thanks man! Ya, I don't like that it needs so much RAM, but its because elasticsearch is a BEAST!!!! I might try and make it so that malice could use something like postgres for people who don't need the whole full text search/kibana (Splunk like) experience.

rufftruffles commented 5 years ago

Thanks man! Ya, I don't like that it needs so much RAM, but its because elasticsearch is a BEAST!!!! I might try and make it so that malice could use something like postgres for people who don't need the whole full text search/kibana (Splunk like) experience.

That would be great man, malice would be able to cater people who don't have enough system resources!

btw, any update on kaspersky plugin?

blacktop commented 5 years ago

Another thing you might try is manually start elasticsearch with this -e ES_JAVA_OPTS="-Xms2g -Xmx2g" then elasticsearch might not try to grab ALL the ram it can

blacktop commented 5 years ago

btw, any update on kaspersky plugin?

I tried getting a demo install so I could figure out how to install it/make a plugin, but I could never find the linux-server installer and the customer support people were ignoring me. If someone could find or "share" with me the linux-server installer I could figure it out 😉

rufftruffles commented 5 years ago

btw, any update on kaspersky plugin?

I tried getting a demo install so I could figure out how to install it/make a plugin, but I could never find the linux-server installer and the customer support people were ignoring me. If someone could find or "share" with me the linux-server installer I could figure it out

I think this is what you're looking for: https://support.kaspersky.com/linux_file80#downloads

https://products.s.kaspersky-labs.com/multilanguage/file_servers/kavlinuxserver8.0/kav4fs_8.0.4-312_i386.deb https://products.s.kaspersky-labs.com/multilanguage/file_servers/kavlinuxserver8.0/kav4fs-8.0.4-312.i386.rpm

blacktop commented 5 years ago

Hmmmm I'll try again, but I remember I wasted a whole weekend a while back trying to find an installer that would work/(AND get updated signatures) and failed.

rufftruffles commented 5 years ago

Hmmmm I'll try again, but I remember I wasted a whole weekend a while back trying to find an installer that would work/(AND get updated signatures) and failed.

Great! Do let me know if you need any server/vm for your testing and research, I'd gladly donate for this awesome project (I have a hosting company). Let me know :)

blacktop commented 5 years ago

Thanks!! I appreciate it, and thank you for helping me find these bugs, I don't know how long there have existed and no one told me. People were probably just trying it, seeing it fail and then thinking... hm this is a piece of crap, and then moving on. So many thanks to you for working with me to figure this out! 👍

Feel free to close this issue. I'll create a FAQ from some of the elasticsearch issues soon.

rufftruffles commented 5 years ago

Thanks!! I appreciate it, and thank you for helping me find these bugs, I don't know how long there have existed and no one told me. People were probably just trying it, seeing it fail and then thinking... hm this is a piece of crap, and then moving on. So many thanks to you for working with me to figure this out!

Feel free to close this issue. I'll create a FAQ from some of the elasticsearch issues soon.

Perfect! I'll create a simple bash script to setup everything on a fresh install and post it here in a day or two, so people don't move on if they get the issues!

Do let me know about the donation offer :)

rufftruffles commented 5 years ago

One last thing, how do I update av keys?

blacktop commented 5 years ago

https://github.com/malice-plugins/avast/blob/master/README.md#use-your-own-license-key

blacktop commented 5 years ago

https://github.com/malice-plugins/avira/blob/master/docs/license.md

blacktop commented 5 years ago

I usually update the trial key on the avast image once a month, but often forget to. I'll do it now

rufftruffles commented 5 years ago

I usually update the trial key on the avast image once a month, but often forget to. I'll do it now

thank you!