Elasticsearch AccessDeniedException

[tw4l]

Hi folks,

I'm attempting to get a development environment up and running (on a Macbook Pro running macOS 10.14.5). I'm able to run Bootstrap successfully and Elasticsearch has enough virtual memory, but the Dashboard and Elasticsearch containers are still down due to what seems to be a permissions error (java.nio.file.AccessDeniedException) in the Elasticsearch container (built from the docker.elastic.co/elasticsearch/elasticsearch:6.5.4 image, per the un-changed docker-compose.yml).

Relevant section of the Elasticsearch log here:

elasticsearch_1                  | Caused by: java.lang.IllegalStateException: Failed to create node environment
elasticsearch_1                  |     at org.elasticsearch.node.Node.<init>(Node.java:299) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     ... 6 more
elasticsearch_1                  | Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
elasticsearch_1                  |     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
elasticsearch_1                  |     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
elasticsearch_1                  |     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
elasticsearch_1                  |     at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:385) ~[?:?]
elasticsearch_1                  |     at java.nio.file.Files.createDirectory(Files.java:689) ~[?:?]
elasticsearch_1                  |     at java.nio.file.Files.createAndCheckIsDirectory(Files.java:796) ~[?:?]
elasticsearch_1                  |     at java.nio.file.Files.createDirectories(Files.java:782) ~[?:?]
elasticsearch_1                  |     at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:203) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.4.jar:6.5.4]
elasticsearch_1                  |     ... 6 more

Is this something you've encountered before? Any hints with debugging? I'm a bit confused because all of the issues I'm finding seem to relate to external volumes, not internal volumes. Thanks!

[sevein]

I never had that issue myself although it's been a while since I've used Docker for Mac. I know @cole uses it on a daily basis though!

[cole]

Huh, it's working fine for me right now.

Docker desktop 2.1.0.1 (stable) Docker engine 19.03.1 Docker compose 1.24.1

If you're on the same versions, I would probably try removing the container/volume and recreating, and if that doesn't work, specifying :rw on the volume mount. I don't really understand why it would behave differently though.

[tw4l]

Hi @cole and @sevein - thanks for the help! In the end the issue turned out to be a memory allocation issue. I'll close this for now, but in case anyone else runs into this issue later working with Docker for Mac, the steps described by Charith here worked for me to reset vm.max_map_count, and now all of the containers are starting properly.

[ross-spencer]

Sorry I wasn't following this one more closely @timothyryanwalsh I think that solution might be what we've documented here. There's a lot to try and keep in one's head. Glad you're up and running though.

[tw4l]

Thanks, Ross! It's the same issue as what you link to, but the steps to make the change are a bit different on Docker for Mac. And a bit elusive, as it seems I may have been a bit premature in declaring this fixed... will continue to look into it and update here when I have it figured out.

[cole]

I'm not sure if that setting works anymore for Docker for Mac? I have 4GB allocated in docker desktop (Preferences -> Advanced).

[tw4l]

I checked and already had 8GB allocated to Docker, so I think maybe this isn't a memory issue after all.

Oddly, I renamed each instance of elasticsearch_data in the docker-compose.yml file to esdata and suddenly now everything is working and I have a functioning Dashboard. Tbh I don't know why this worked (as far as I know there's no rule against underscores in volume names, and the mysql named volume with an underscore is working fine), but I'll take it haha.

Thanks again @cole @ross-spencer @sevein for your help!