search

komoot / photon

an open source geocoder for openstreetmap data
Apache License 2.0
1.92k stars 281 forks source link

Startup fails due to self built index is in recovering state #716

Closed sibbl closed 1 year ago

sibbl commented 1 year ago

Hi, first of all thanks for that great project!

I'm building my own index from scratch and unfortunately I'm having some issues when starting Photon with this index. It seems that according to ElasticSearch it is in "recovering" state. However, it feels like it cannot complete recovering due to Photon crashing 10s after "ES cluster is now ready" appears, so this is basically an endless loop which I am not able to fix...

Is there any recommendation about how to let ES finish recovering or - even better - how to avoid the index being in the recovering state right on startup?

Below is my full stack trace.

Thanks in advance for any help!

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2023-02-02 16:30:28,770 [main] WARN  org.elasticsearch.node.Node - version [5.6.16-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
2023-02-02 16:30:37,153 [main] INFO  de.komoot.photon.elasticsearch.Server - started elastic search node
2023-02-02 16:30:37,153 [main] INFO  de.komoot.photon.App - Make sure that the ES cluster is ready, this might take some time.
2023-02-02 16:31:07,168 [main] INFO  de.komoot.photon.App - ES cluster is now ready.
Exception in thread "main" NoShardAvailableActionException[No shard available for [get [photon][place][DATABASE_PROPERTIES]: routing [null]]]; nested: RemoteTransportException[[RhpYFsP][127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:196)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1300(TransportSingleShardAction.java:123)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:252)
    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1085)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1189)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1167)
    at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:673)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:660)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: RemoteTransportException[[RhpYFsP][127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
Caused by: [photon/3a877LrxTseV62PeKN0zBA][[photon][0]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]
    at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1104)
    at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:615)
    at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:154)
    at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:82)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:294)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:287)
    at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)