Default docker-machine has insufficient RAM to set up docker-compose

[lvh]

Currently, the Kafka JVM attempts to pre-allocate 1024MB all by itself. This is problematic because the standard docker-machine VM only has 1024MB total, so the malloc fails, so the Kafka container falls over immediately. This is unrelated to #45 as far as I can tell (other than that it's also about Kafka).

lvh@zygalski ~/P/r/desdemona (master)> docker-machine create --driver virtualbox rax2                                                                                                               09:18:22
Running pre-create checks...
(rax2) Default Boot2Docker ISO is out-of-date, downloading the latest release...
(rax2) Latest release for github.com/boot2docker/boot2docker is v1.10.2
(rax2) Downloading /Users/lvh/.docker/machine/cache/boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v1.10.2/boot2docker.iso...
(rax2) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
Creating machine...
(rax2) Copying /Users/lvh/.docker/machine/cache/boot2docker.iso to /Users/lvh/.docker/machine/machines/rax2/boot2docker.iso...
(rax2) Creating VirtualBox VM...
(rax2) Creating SSH key...
(rax2) Starting the VM...
(rax2) Check network to re-create if needed...
(rax2) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env rax2
lvh@zygalski ~/P/r/desdemona (master)> VboxManage showvminfo rax2 | grep Memory                                                                                                                     09:20:52
Memory size:     1024MB

One way to mitigate this immediately would be to document it so that people can just adjust their VM. Medium-term it would make sense to get it to try and allocate less memory, since I'm assuming lots of people will be using the default. Longer-term it maybe useful to make that amount of memory configurable; I could certainly see how a production setup would want more than 1024MB.

[lvh]

cc @sirsean -- any opinions on that last paragraph?

[lvh]

FYI, this looks like:

kafka_1     | # There is insufficient memory for the Java Runtime Environment to continue.
kafka_1     | # Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory.

[sirsean]

I'm 100% on board with documenting that you need at least 2GB to run this thing, and doing that immediately.

I also very much like the idea of figuring out how to make memory use configurable -- when we go to production we're going to want to use more. But that's kind of a separate step from creating the larger VM's for the production environment.

And I don't know how valuable it is to figure out what we have to do to Kafka to get it to run in less RAM.

[lvh]

Okiedokie. #63 documents. It doesn't seem like Kafka would need 1G RAM for the basic things we're asking it to do in most dev setups. Obviously once you start throwing load at it... This would mean changing the Xms JVM parameter.

The other thing to keep in mind is that unless you set Xmx, lowering this limit does not affect how much memory the JVM will maximally try to allocate.

[lvh]

I think that simply solving #65 solves this for now. A bigger minimum preallocation for production Kafka setups, seems to be a bridge we should cross when we get there. Are you OK with closing this ticket and just leaving it with #65, @sirsean ?

[sirsean]

Yeah, that's fine with me. :+1: