Nodes are not running

[jjacobson93]

Pulled the latest Docker image, ran it with the recommended command, and looking at the memsql-ops panel, the nodes are either not connected or are not running. There are no errors on stdout/stderr either.

*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/rc.local...
*** Booting runit daemon...
*** Runit started as PID 10
Sep 30 01:57:49 9bc9aa92b6ee syslog-ng[17]: syslog-ng starting up; version='3.5.3'

[carlsverre]

Did you run it by mounting a volume? We have a bug in the current iteration that causes it to fail when the volume is mounted via nfs (for example if you are running the quickstart container on a mac in boot2docker or docker-machine). We have a fix for this that will be on our docs tomorrow. For the time being either don't use a volume (for testing) or use another container (as a data container) for the volume. For example:

docker create --name memsql_data memsql/quickstart /bin/true
docker run -d --name memsql -p 3306:3306 -p 9000:9000 --volumes-from memsql_data memsql/quickstart

[jjacobson93]

I'm running on a Mac with docker-machine. Not using any volume or using another container results in the same issue. I'll check back tomorrow

[carlsverre]

Interesting - thanks for the update.

The first thing I am curious about is why you have 3 MemSQL nodes in the attached screenshot. Our docker-quickstart container only includes two MemSQL nodes so it seems that you launched a third.

Also, can you use docker exec to access the instance and check the memsql logs? For example:

~ $ docker run -d --name memsql -p 3306:3306 -p 9000:9000 memsql/quickstart
0aa3eee01a9483c4475b3e28b68ed014bc79bdf7a3968da369394a4aeca997df
~ $ docker exec -it memsql /bin/bash
root@0aa3eee01a94:/# cd /memsql/master/tracelogs
root@0aa3eee01a94:/memsql/master/tracelogs# cat memsql.log
64 2015-09-27 20:05:42 INFO: Log opened
...

If the logs are empty also check the MemSQL Ops logs at /memsql-ops/memsql-ops.log.

[carlsverre]

Also, can you pull down memsql/quickstart:latest again? I just pushed a version with some tweaks to it.

[jjacobson93]

Ah looks like here's the problem:

79 2015-09-30 16:20:46 INFO: Log opened
00700676 2015-09-30 16:20:47  INFO: Initializing Open SSL
00702005 2015-09-30 16:20:47  INFO: MemSQL version hash: f8b720e1ac467b30e2b96f7fd250820bc1a2aca5 (Wed Sep 23 21:57:54 2015 +0000)
00895516 2015-09-30 16:20:47  INFO: ./memsqld: ready for connections.
00895562 2015-09-30 16:20:47  INFO: Version:  '4.1'  Edition:  'community'  Socket:  '/memsql-ops/installs/master-3306/data/memsql.sock'  Port:  '3306'
00896697 2015-09-30 16:20:47  INFO: System Information: sockets (1), physical cores (8), virtual cores (8), model name (Intel(R) Core(TM) i7-3635QM CPU @ 2.40GHz), memory (8363749376), uname (Linux 2338b11efbe1 4.0.9-boot2docker #1 SMP Thu Aug 13 03:05:44 UTC 2015 x86_64)
01053411 2015-09-30 16:20:48  INFO: starting up with aggregator id 0
01053522 2015-09-30 16:20:48  INFO: Setting aggregator id to 1
01056470 2015-09-30 16:20:48  INFO: Master aggregator has started with ID=1
01056666 2015-09-30 16:20:48  INFO: Completed recovery of system databases.  Beginning recovery of user databases.
01057750 2015-09-30 16:20:48  INFO: Used cluster capacity: 7179, Maximum cluster capacity: 18446744073709551615
03253246 2015-09-30 16:20:50  INFO: Adding new leaf node root@127.0.0.1:3307
03606419 2015-09-30 16:20:50  INFO: New node 2 (127.0.0.1:3307)
03645656 2015-09-30 16:20:50  WARN: ReplicationMaster[sharding]: Sync slave '127.0.0.1:3307/sharding' got synchronized with master 'sharding'.
03791715 2015-09-30 16:20:50  INFO: Done Adding new leaf node root@127.0.0.1:3307
35 2015-10-01 02:45:22 INFO: Log opened
00000065 2015-10-01 02:45:22 FATAL: This machine does not have enough CPU cores to run MemSQL. The minimum required number is 4, and this machine has 1 cores

I was able to increase the amount of CPUs on my docker host and everything seems to be working now.

[jjacobson93]

And yes, I tried adding a third node before.

[carlsverre]

Ah, good find! I am updating our check-system script which is included with memsql/quickstart to include a core check in order to make this easier to debug.

[GuiSim]

I ran into what might be this issue again on Mac OS X.

Digging into the docker's logs, I found

2015-11-27 16:22:31,080 [ERROR] memsql_platform.jobs.engine: Failed to execute job: Failed to start MemSQL: This machine does not have enough CPU cores to run MemSQL. The minimum required number is 4, and this machine has 1 cores

The system check was successful.

[GuiSim]

Increasing the docker-machine's CPU count to 4 solved this.