xpack-ml related error on startup of 6.3.0 image

[rocketraman]

Using the 6.3.0 images, hash b16d5e2a8db4, I am getting an error on startup inside a Kubernetes environment:

[2018-06-17T03:58:42,113][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-master-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Cannot run program "/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller": error=2, No such file or directory
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.0.jar:6.3.0]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
Caused by: org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Cannot run program "/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller": error=2, No such file or directory
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:168) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]
    ... 6 more
Caused by: java.io.IOException: Cannot run program "/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_151]
    at org.elasticsearch.bootstrap.Spawner.spawnNativeController(Spawner.java:118) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Spawner.spawnNativeControllers(Spawner.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:166) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]
    ... 6 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method) ~[?:1.8.0_151]
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) ~[?:1.8.0_151]
    at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[?:1.8.0_151]
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_151]
    at org.elasticsearch.bootstrap.Spawner.spawnNativeController(Spawner.java:118) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Spawner.spawnNativeControllers(Spawner.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:166) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]
    ... 6 more

Looking into the container, this file actually does exist, but the No such file or directory error is raised by the container OS when /elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller is executed, due presumably to missing libraries (as reported by ldd).

This error seems to be thrown regardless of the xpack.ml.enabled=false setting in the upstream images. From the stack trace, it does look like this is happening early in the boot process and before any config files are even loaded. However, if I'm right I guess this should be happening to everyone using 6.3.0 and so far I'm the only one reporting it, so that is a bit of a mystery...

Also note: this did not happen on 6.2.4.

[pires]

Please see https://github.com/pires/docker-elasticsearch/blob/master/run.sh#L50-L51. I am unable to reproduce this.

[pires]

I'm going to rebuild docker-elasticsearch and docker-elasticsearch-kubernetes.

[pires]

New SHA: dcd3e9db3d2c. Please try it and let me know.

[rocketraman]

Please see https://github.com/pires/docker-elasticsearch/blob/master/run.sh#L50-L51.

You might want to use $BASE to be consistent e.g.:

rm -rf $BASE/modules/x-pack/x-pack-ml

Also, on my local image, I removed only the platform files, which was sufficient to solve the problem:

rm -rf $BASE/modules/x-pack/x-pack-ml/platform/linux-x86_64

I am unable to reproduce this.

I have no clue why this is happening only for me.

[pires]

Have you tried the new SHA?

New SHA: dcd3e9db3d2c. Please try it and let me know.

[ploef]

Hi,

Using the ES 6.4.0 and having the exact same problem. ES 6.2.4 is working ok.

regards, Peter

[kasunt84]

@ploef with ES 6.4.x you need to do rm -rf $BASE/modules/x-pack-ml

[pires]

🤦‍♂️ sorry i didn't validate this. I'm going to re-tag 6.4.0 after testing what @kasunt84 suggested.

[kasunt84]

thanks @pires forgot to open a PR for it.

[pires]

@kasunt84 can you try and pull the image again? The sha is f349c6607a17.

[kasunt84]

@pires thanks. appears to be working for me

[ploef]

Is working for me too. Thanks!