Elasticsearch required increased file descriptor limits

[philrz]

Describe the bug

When I tried to do my first run of malice using a command-line from the examples/lookup doc, the Elasticsearch launched by Docker failed to start. The error message:

ERROR: [1] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

To Reproduce

1. Start with a fresh Ubuntu 18.04.1 environment
2. Download/install release v0.3.28
3. Invoke malice, such as with malice lookup 6fe80e56ad4de610304bab1675ce84d16ab6988e

Environment (please complete the following information):

Ubuntu 18.04.1

Output of docker version:

# docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.4
 Git commit:        e68fc7a
 Built:             Mon Oct  1 14:25:31 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       e68fc7a
  Built:            Mon Oct  1 14:25:33 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

# docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 24
Server Version: 18.06.1-ce
Storage Driver: aufs
 Root Dir: /var/snap/docker/common/var-lib-docker/aufs
 Backing Filesystem: extfs
 Dirs: 151
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: N/A (expected: 69663f0bd4b60df09991c08812a60108003fa340)
init version: 949e6fa (expected: fec3683)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-43-generic
Operating System: Ubuntu Core 16
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.79GiB
Name: ubuntu18
ID: VFAG:SG62:7RX3:6MJX:EQ2B:MAXZ:7P2T:V6CC:UYFU:X4BO:GEVF:DVGM
Docker Root Dir: /var/snap/docker/common/var-lib-docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 35
 Goroutines: 56
 System Time: 2019-01-07T17:51:43.77509935Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional context I've been able to work around this problem via two steps:

1. Added to /etc/security/limits.conf:

   *       soft    nofile      262144
   *       hard    nofile      262144

2. Launched Elasticsearch manually before running malice, using a command-line that increases the file descriptor limits:

   # docker run --init -d --name malice-elastic --ulimit nofile=262144:262144 -p 9200:9200 -e ES_JAVA_OPTS="-Xms2g -Xmx2g" malice/elasticsearch:6.5

I'm not sure if this is the best way to address the issue.

[blacktop]

Hi @philrz !!!

Yeah, I think me trying to "bundle" elasticsearch with malice is causing a lot of head aches because a lot of people's issues with running malice are with elasticsearch.

• https://github.com/maliceio/malice#known-issues-warning
• https://github.com/maliceio/malice/blob/master/docs/KnownBugs.md

So maybe I'll change the steps to get elasticsearch up and running first by yourself and THEN use malice 😁

I do ❤️ elasticsearch though.... 🤔

[philrz]

Thanks for the quick response. Yeah, I saw the list of known issues with ES, but since mine wasn't in there, figured I'd point it out for completeness.

I agree that including a brief ES cookbook and calling it a prerequisite might be easier than trying to bundle it. Thankfully with all the Elastic docs, containers, articles, etc. it'd probably not be hard to point to simple "hello world" level steps to get something basic running and not slow down beginners too much.

BTW, I tried to ping you on Keybase because I really like the look of malice but I have a couple general questions I was hoping to ask you about the project. I'm guessing it's all stuff you probably covered at your Blackhat 2018 presentation, but unfortunately I wasn't able to attend that. :-/ I'm local to the bay area and I sense you may be local as well, so if I can bribe you with free lunch, just name the place. :)

[blacktop]

Do you mean your solution was different?

It seems like sudo sysctl -w vm.max_map_count=262144 would have fixed it? If not, then I apologize, I see SO many issues created by people with elasticsearch issues that I assumed it was the "main" issue that I linked to.

I'll hit you up on keybase soon!

[philrz]

Yeah, it's separate issues. You're right that the max_map_count setting is important, but I knew to up my sysctl there because I've been bitten by it in other ES contexts. The file descriptors limit is a separate roadblock that I hadn't run into before, so I did some searches to come up with my workaround.

I'm not sure why nobody else had reported it before me. Maybe most others are running on other platforms and this limit doesn't affect them? Just a guess. I've only tried on Ubuntu 18.04.1 thus far.