search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
984 stars 24.82k forks source link

`path.data` can no longer be a symlink #85701

Closed DaveCTurner closed 2 years ago

DaveCTurner commented 2 years ago

Elasticsearch Version

8.0.0 and up

Installed Plugins

N/A

Java Version

bundled

OS Version

Linux

Problem Description

In versions prior to 8.0.0 it was ok if path.data was a symlink (to a directory) but since 8.0.0 Elasticsearch will fail to start up unless path.data is genuinely a directory.

I think this regression was introduced in this change: we used to create ${path.data}/nodes/0 but now we create ${path.data}, and Files#createDirectories is ok if one of the ancestors of the target path is a symlink but it will fail if the target path itself is not a genuine directory.

Steps to Reproduce

$ tar xf ~/Downloads/elasticsearch-8.0.0.tar.gz
$ mkdir elasticsearch-8.0.0/actual-data
$ ln -sf actual-data elasticsearch-8.0.0/data
$ elasticsearch-8.0.0/bin/elasticsearch

Logs (if relevant)

[2022-04-05T15:38:30,595][INFO ][o.e.n.Node               ] [MacBook-Pro.lan] version[8.0.0], pid[21391], build[default/tar/1b6a7ece17463df5ff54a3e1302d825889aa1161/2022-02-03T16:47:57.507843096Z], OS[Mac OS X/10.15.7/x86_64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]
[2022-04-05T15:38:30,599][INFO ][o.e.n.Node               ] [MacBook-Pro.lan] JVM home [/Users/davidturner/discuss/301126/elasticsearch-8.0.0/jdk.app/Contents/Home], using bundled JDK [true]
[2022-04-05T15:38:30,599][INFO ][o.e.n.Node               ] [MacBook-Pro.lan] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/var/folders/y9/4hv0mzbj4h7b41jw89hs0n800000gn/T/elasticsearch-5927348957417933086, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/Users/davidturner/discuss/301126/elasticsearch-8.0.0, -Des.path.conf=/Users/davidturner/discuss/301126/elasticsearch-8.0.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-04-05T15:38:31,871][WARN ][stderr                   ] [MacBook-Pro.lan] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2022-04-05T15:38:31,872][WARN ][stderr                   ] [MacBook-Pro.lan] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2022-04-05T15:38:31,872][WARN ][stderr                   ] [MacBook-Pro.lan] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-04-05T15:38:32,757][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [aggs-matrix-stats]
[2022-04-05T15:38:32,757][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [analysis-common]
[2022-04-05T15:38:32,758][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [constant-keyword]
[2022-04-05T15:38:32,758][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [frozen-indices]
[2022-04-05T15:38:32,758][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [ingest-common]
[2022-04-05T15:38:32,758][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [ingest-geoip]
[2022-04-05T15:38:32,758][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [ingest-user-agent]
[2022-04-05T15:38:32,758][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [kibana]
[2022-04-05T15:38:32,759][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [lang-expression]
[2022-04-05T15:38:32,759][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [lang-mustache]
[2022-04-05T15:38:32,759][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [lang-painless]
[2022-04-05T15:38:32,759][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [legacy-geo]
[2022-04-05T15:38:32,759][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [mapper-extras]
[2022-04-05T15:38:32,759][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [mapper-version]
[2022-04-05T15:38:32,759][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [parent-join]
[2022-04-05T15:38:32,762][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [percolator]
[2022-04-05T15:38:32,762][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [rank-eval]
[2022-04-05T15:38:32,762][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [reindex]
[2022-04-05T15:38:32,763][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [repositories-metering-api]
[2022-04-05T15:38:32,763][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [repository-azure]
[2022-04-05T15:38:32,763][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [repository-encrypted]
[2022-04-05T15:38:32,763][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [repository-gcs]
[2022-04-05T15:38:32,763][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [repository-s3]
[2022-04-05T15:38:32,764][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [repository-url]
[2022-04-05T15:38:32,764][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [runtime-fields-common]
[2022-04-05T15:38:32,764][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [search-business-rules]
[2022-04-05T15:38:32,764][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [searchable-snapshots]
[2022-04-05T15:38:32,765][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [snapshot-based-recoveries]
[2022-04-05T15:38:32,765][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [snapshot-repo-test-kit]
[2022-04-05T15:38:32,765][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [spatial]
[2022-04-05T15:38:32,765][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [transform]
[2022-04-05T15:38:32,765][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [transport-netty4]
[2022-04-05T15:38:32,766][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [unsigned-long]
[2022-04-05T15:38:32,766][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [vector-tile]
[2022-04-05T15:38:32,766][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [vectors]
[2022-04-05T15:38:32,766][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [wildcard]
[2022-04-05T15:38:32,766][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-aggregate-metric]
[2022-04-05T15:38:32,767][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-analytics]
[2022-04-05T15:38:32,767][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-async]
[2022-04-05T15:38:32,767][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-async-search]
[2022-04-05T15:38:32,767][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-autoscaling]
[2022-04-05T15:38:32,767][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-ccr]
[2022-04-05T15:38:32,767][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-core]
[2022-04-05T15:38:32,767][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-data-streams]
[2022-04-05T15:38:32,767][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-deprecation]
[2022-04-05T15:38:32,768][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-enrich]
[2022-04-05T15:38:32,768][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-eql]
[2022-04-05T15:38:32,768][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-fleet]
[2022-04-05T15:38:32,768][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-graph]
[2022-04-05T15:38:32,768][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-identity-provider]
[2022-04-05T15:38:32,768][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-ilm]
[2022-04-05T15:38:32,768][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-logstash]
[2022-04-05T15:38:32,768][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-ml]
[2022-04-05T15:38:32,768][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-monitoring]
[2022-04-05T15:38:32,769][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-ql]
[2022-04-05T15:38:32,769][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-rollup]
[2022-04-05T15:38:32,769][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-security]
[2022-04-05T15:38:32,769][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-shutdown]
[2022-04-05T15:38:32,769][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-sql]
[2022-04-05T15:38:32,769][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-stack]
[2022-04-05T15:38:32,769][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-text-structure]
[2022-04-05T15:38:32,770][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-voting-only-node]
[2022-04-05T15:38:32,770][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] loaded module [x-pack-watcher]
[2022-04-05T15:38:32,770][INFO ][o.e.p.PluginsService     ] [MacBook-Pro.lan] no plugins loaded
[2022-04-05T15:38:32,823][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [MacBook-Pro.lan] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchException: failed to bind service
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-8.0.0.jar:8.0.0]
    at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-8.0.0.jar:8.0.0]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
    at org.elasticsearch.node.Node.<init>(Node.java:1005) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-8.0.0.jar:8.0.0]
    ... 6 more
Caused by: java.nio.file.FileAlreadyExistsException: /Users/davidturner/discuss/301126/elasticsearch-8.0.0/data
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:94) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
    at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:398) ~[?:?]
    at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]
    at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]
    at java.nio.file.Files.createDirectories(Files.java:753) ~[?:?]
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:261) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.node.Node.<init>(Node.java:388) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-8.0.0.jar:8.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-8.0.0.jar:8.0.0]
    ... 6 more
uncaught exception in thread [main]
org.elasticsearch.ElasticsearchException: failed to bind service
Likely root cause: java.nio.file.FileAlreadyExistsException: /Users/davidturner/discuss/301126/elasticsearch-8.0.0/data
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:398)
    at java.base/java.nio.file.Files.createDirectory(Files.java:700)
    at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
    at java.base/java.nio.file.Files.createDirectories(Files.java:753)
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:261)
    at org.elasticsearch.node.Node.<init>(Node.java:388)
    at org.elasticsearch.node.Node.<init>(Node.java:277)
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
    at org.elasticsearch.cli.Command.main(Command.java:77)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
elasticmachine commented 2 years ago

Pinging @elastic/es-core-infra (Team:Core/Infra)