search

molgenis / molgenis-app-palga

Molgenis app for PALGA
GNU Lesser General Public License v3.0
0 stars 9 forks source link

Palga server sometimes fails to boot. #78

Open fdlk opened 6 years ago

fdlk commented 6 years ago

How to Reproduce

Check palga boot logs on molgenis68.

Expected behavior

Boot succeeds every time

Observed behavior

INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
16:04:00.127 [localhost-startStop-1] INFO  o.m.d.e.f.EmbeddedElasticSearchServiceFactory - Embedded elasticsearch server started, data path=[/data/.molgenis/palga/data]
16:04:32.148 [localhost-startStop-1] WARN  o.m.data.mysql.MysqlRepository - Table for entity tags already exists. Skipping creation
16:04:33.650 [localhost-startStop-1] WARN  o.m.data.mysql.MysqlRepository - Table for entity packages already exists. Skipping creation
16:04:33.744 [localhost-startStop-1] WARN  o.s.w.c.s.AnnotationConfigWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'elasticSearchRepositoryRegistrator' defined in file [/srv/molgenis/apache-tomcat-7.0.78/webapps/ROOT/WEB-INF/classes/org/molgenis/palga/elasticsearch/ElasticSearchRepositoryRegistrator.class]: Unsatisfied dependency expressed through constructor argument with index 7 of type [org.molgenis.data.mysql.MysqlRepositoryCollection]: : Error creating bean with name 'mysqlRepositoryCollection' defined in class path resource [org/molgenis/data/mysql/MySqlConfiguration.class]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.molgenis.data.mysql.MysqlRepositoryCollection org.molgenis.data.mysql.MySqlConfiguration.mysqlRepositoryCollection()] threw exception; nested exception is org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mysqlRepositoryCollection' defined in class path resource [org/molgenis/data/mysql/MySqlConfiguration.class]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.molgenis.data.mysql.MysqlRepositoryCollection org.molgenis.data.mysql.MySqlConfiguration.mysqlRepositoryCollection()] threw exception; nested exception is org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:747) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:185) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1115) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:706) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403) [spring-web-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306) [spring-web-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106) [spring-web-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5116) [catalina.jar:7.0.78]
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634) [catalina.jar:7.0.78]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) [catalina.jar:7.0.78]
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) [catalina.jar:7.0.78]
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875) [catalina.jar:7.0.78]
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) [catalina.jar:7.0.78]
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092) [catalina.jar:7.0.78]
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984) [catalina.jar:7.0.78]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mysqlRepositoryCollection' defined in class path resource [org/molgenis/data/mysql/MySqlConfiguration.class]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.molgenis.data.mysql.MysqlRepositoryCollection org.molgenis.data.mysql.MySqlConfiguration.mysqlRepositoryCollection()] threw exception; nested exception is org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:597) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1095) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:990) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1021) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:964) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:862) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:811) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:739) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    ... 28 common frames omitted
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.molgenis.data.mysql.MysqlRepositoryCollection org.molgenis.data.mysql.MySqlConfiguration.mysqlRepositoryCollection()] threw exception; nested exception is org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:188) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:586) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    ... 41 common frames omitted
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233) ~[elasticsearch-1.4.4.jar:na]
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179) ~[elasticsearch-1.4.4.jar:na]
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565) ~[elasticsearch-1.4.4.jar:na]
    ... 3 common frames omitted
16:04:34.561 [localhost-startStop-1] INFO  o.m.d.e.f.EmbeddedElasticSearchServiceFactory - Elastic search server stopped
16:04:34.569 [localhost-startStop-1] ERROR o.s.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'elasticSearchRepositoryRegistrator' defined in file [/srv/molgenis/apache-tomcat-7.0.78/webapps/ROOT/WEB-INF/classes/org/molgenis/palga/elasticsearch/ElasticSearchRepositoryRegistrator.class]: Unsatisfied dependency expressed through constructor argument with index 7 of type [org.molgenis.data.mysql.MysqlRepositoryCollection]: : Error creating bean with name 'mysqlRepositoryCollection' defined in class path resource [org/molgenis/data/mysql/MySqlConfiguration.class]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.molgenis.data.mysql.MysqlRepositoryCollection org.molgenis.data.mysql.MySqlConfiguration.mysqlRepositoryCollection()] threw exception; nested exception is org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mysqlRepositoryCollection' defined in class path resource [org/molgenis/data/mysql/MySqlConfiguration.class]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.molgenis.data.mysql.MysqlRepositoryCollection org.molgenis.data.mysql.MySqlConfiguration.mysqlRepositoryCollection()] threw exception; nested exception is org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:747) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:185) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1115) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:706) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403) ~[spring-web-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306) ~[spring-web-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106) [spring-web-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5116) [catalina.jar:7.0.78]
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634) [catalina.jar:7.0.78]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) [catalina.jar:7.0.78]
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) [catalina.jar:7.0.78]
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875) [catalina.jar:7.0.78]
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) [catalina.jar:7.0.78]
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092) [catalina.jar:7.0.78]
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984) [catalina.jar:7.0.78]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mysqlRepositoryCollection' defined in class path resource [org/molgenis/data/mysql/MySqlConfiguration.class]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.molgenis.data.mysql.MysqlRepositoryCollection org.molgenis.data.mysql.MySqlConfiguration.mysqlRepositoryCollection()] threw exception; nested exception is org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:597) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1095) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:990) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1021) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:964) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:862) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:811) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:739) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    ... 28 common frames omitted
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.molgenis.data.mysql.MysqlRepositoryCollection org.molgenis.data.mysql.MySqlConfiguration.mysqlRepositoryCollection()] threw exception; nested exception is org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:188) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:586) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
    ... 41 common frames omitted
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233) ~[elasticsearch-1.4.4.jar:na]
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179) ~[elasticsearch-1.4.4.jar:na]
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565) ~[elasticsearch-1.4.4.jar:na]
    ... 3 common frames omitted
Nov 27, 2017 4:04:34 PM org.apache.catalina.core.StandardContext startInternal
fdlk commented 6 years ago

Some observations:

dennishendriksen commented 6 years ago

@fdlk I think the relevant part of the stack trace is Failed to execute phase [query_fetch], all shards failed which implies that a query was executed before any of the shards (we only have one) are ready.

fdlk commented 6 years ago

@dennishendriksen yes, think so too, but I cannot exactly find the line where that happens. Ton of common frames but who's doing what when the actual exception happens? Other question is what makes the difference with before. Server park busier? More data in the index? Faster JVM?

fdlk commented 6 years ago

Again failed to boot on Jan 29, 2018.

fdlk commented 6 years ago

Again failed to boot on Feb 5, 2018

mswertz commented 6 years ago

we aim to bring palga server into standard molgenis

dennishendriksen commented 6 years ago

@mswertz Lets close once we updated PALGA to use the latest MOLGENIS and keep the issue open so we can find it when the current PALGA fails booting.