search

vespa-engine / vespa

AI + Data, online. https://vespa.ai
https://vespa.ai
Apache License 2.0
5.8k stars 604 forks source link

Cannot ReStart crashed Vespa #18576

Closed sirganya closed 3 years ago

sirganya commented 3 years ago

I have a Vespa instance running on CentOS that was working for a few months and crashed. I have one schema called "stacks". When I try restarting it I get this error:

java.lang.RuntimeException: Document type 'stack' not found in application package

Have I corrupted my DB

bratseth commented 3 years ago

No, it’s just something wrong with your application package. Perhaps you put stacks.sd in the wrong dir? Or named it something else?

sirganya commented 3 years ago

Should the application be in $VESPA_HOME/var/db/vespa/config_server/serverdb/

I've been using curl to prepare and deploy so far.

bratseth commented 3 years ago

You really get this when you restart the config server, not when you deploy?

sirganya commented 3 years ago

Sorry, my point was that as I’ve only used a curl operation to deploy an application I don’t know where to look on the sever for the faulty application to check if it’s there as you suggested. I don’t know where Vespa puts them.

bratseth commented 3 years ago

Yes, I understand, but just trying to confirm that you really get this on restart of the config server, not on redeploy. The application package is stored in ZooKeeper so it is not expected that anything happens to that data after you deploy.

sirganya commented 3 years ago

Yes, it's on restart. After it crashed I triggered a reinstall of vespa accidentally. I had customised the AWS multinode set up script as a template to install it without docker and mistakenly ran it instead of a startup script. I'm new to Vespa and inexperienced with JVMs.

My main hope is to save the documents.

[2021-07-08 16:47:10.224] ERROR : configserver Container.com.yahoo.jdisc.core.StandaloneMain JDisc exiting: Throwable caught: \nexception=\ncom.yahoo.container.di.componentgraph.core.ComponentNode$ComponentConstructorException: Error constructing 'com.yahoo.vespa.config.server.tenant.TenantRepository': null\nCaused by: java.lang.RuntimeException: Could not create all tenants when bootstrapping, failed to create: [default]\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.bootstrapTenants(TenantRepository.java:290)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.<init>(TenantRepository.java:210)\n [centos@ip-172-31-83-99 ~]$

[2021-07-08 16:46:41.287] EVENT : configserver Container.com.yahoo.statistics.Counter count/1 name=configserver.requests value=0 [2021-07-08 16:46:41.287] EVENT : configserver Container.com.yahoo.statistics.Counter count/1 name=configserver.failedRequests value=0 [2021-07-08 16:46:41.287] EVENT : configserver Container.com.yahoo.statistics.Counter count/1 name=procTime value=0 [2021-07-08 16:46:42.604] EVENT : configserver runserver stopped/1 name="java -Xms128m -Xmx2048m -XX:+PreserveFramePointer -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/vespa/var/crash -XX:ErrorFile=/opt/vespa/var/crash/hs_err_pid%p.log -XX:+ExitOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -XX:MaxJavaStackTraceDepth=1000000 --illegal-access=warn --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/sun.security.ssl=ALL-UNNAMED -Djava.io.tmpdir=/opt/vespa/tmp -Djava.library.path=/opt/vespa/lib64 -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.net.client.defaultConnectTimeout=5000 -Dsun.net.client.defaultReadTimeout=60000 -Djavax.net.ssl.keyStoreType=JKS -Djdk.tls.rejectClientInitiatedRenegotiation=true -Djdisc.config.file=/opt/vespa/var/jdisc_container/configserver.properties -Djdisc.export.packages= -Djdisc.cache.path=/opt/vespa/var/vespa/bundlecache/configserver -Djdisc.debug.resources=false -Djdisc.bundle.path=/opt/vespa/lib/jars -Djdisc.logger.enabled=true -Djdisc.logger.level=ALL -Djdisc.logger.tag=jdisc/configserver -Dfile.encoding=UTF-8 -Dzookeeper_log_file_prefix=/opt/vespa/logs/vespa/zookeeper.configserver -cp /opt/vespa/lib/jars/jdisc_core-jar-with-dependencies.jar com.yahoo.jdisc.core.StandaloneMain standalone-container-jar-with-dependencies.jar (pid 2388)" pid=2388 exitcode=6 [2021-07-08 16:47:00.607] EVENT : configserver runserver starting/1 name="java -Xms128m -Xmx2048m -XX:+PreserveFramePointer -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/vespa/var/crash -XX:ErrorFile=/opt/vespa/var/crash/hs_err_pid%p.log -XX:+ExitOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -XX:MaxJavaStackTraceDepth=1000000 --illegal-access=warn --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/sun.security.ssl=ALL-UNNAMED -Djava.io.tmpdir=/opt/vespa/tmp -Djava.library.path=/opt/vespa/lib64 -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.net.client.defaultConnectTimeout=5000 -Dsun.net.client.defaultReadTimeout=60000 -Djavax.net.ssl.keyStoreType=JKS -Djdk.tls.rejectClientInitiatedRenegotiation=true -Djdisc.config.file=/opt/vespa/var/jdisc_container/configserver.properties -Djdisc.export.packages= -Djdisc.cache.path=/opt/vespa/var/vespa/bundlecache/configserver -Djdisc.debug.resources=false -Djdisc.bundle.path=/opt/vespa/lib/jars -Djdisc.logger.enabled=true -Djdisc.logger.level=ALL -Djdisc.logger.tag=jdisc/configserver -Dfile.encoding=UTF-8 -Dzookeeper_log_file_prefix=/opt/vespa/logs/vespa/zookeeper.configserver -cp /opt/vespa/lib/jars/jdisc_core-jar-with-dependencies.jar com.yahoo.jdisc.core.StandaloneMain standalone-container-jar-with-dependencies.jar (pid 2972)" [2021-07-08 16:47:09.397] EVENT : configserver Container.com.yahoo.statistics.Counter count/1 name=configserver.requests value=0 [2021-07-08 16:47:09.398] EVENT : configserver Container.com.yahoo.statistics.Counter count/1 name=configserver.failedRequests value=0 [2021-07-08 16:47:09.398] EVENT : configserver Container.com.yahoo.statistics.Counter count/1 name=procTime value=0 [2021-07-08 16:47:10.665] EVENT : configserver runserver stopped/1 name="java -Xms128m -Xmx2048m -XX:+PreserveFramePointer -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/vespa/var/crash -XX:ErrorFile=/opt/vespa/var/crash/hs_err_pid%p.log -XX:+ExitOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -XX:MaxJavaStackTraceDepth=1000000 --illegal-access=warn --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/sun.security.ssl=ALL-UNNAMED -Djava.io.tmpdir=/opt/vespa/tmp -Djava.library.path=/opt/vespa/lib64 -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.net.client.defaultConnectTimeout=5000 -Dsun.net.client.defaultReadTimeout=60000 -Djavax.net.ssl.keyStoreType=JKS -Djdk.tls.rejectClientInitiatedRenegotiation=true -Djdisc.config.file=/opt/vespa/var/jdisc_container/configserver.properties -Djdisc.export.packages= -Djdisc.cache.path=/opt/vespa/var/vespa/bundlecache/configserver -Djdisc.debug.resources=false -Djdisc.bundle.path=/opt/vespa/lib/jars -Djdisc.logger.enabled=true -Djdisc.logger.level=ALL -Djdisc.logger.tag=jdisc/configserver -Dfile.encoding=UTF-8 -Dzookeeper_log_file_prefix=/opt/vespa/logs/vespa/zookeeper.configserver -cp /opt/vespa/lib/jars/jdisc_core-jar-with-dependencies.jar com.yahoo.jdisc.core.StandaloneMain standalone-container-jar-with-dependencies.jar (pid 2972)" pid=2972 exitcode=6 [2021-07-08 16:47:42.480] EVENT : config-sentinel sentinel.config-sentinel stopping/1 name="config-sentinel" why="Timed out while subscribing to 'cloud.config.sentinel', configid 'hosts/ip-172-31-83-99.ec2.internal'" [2021-07-08 16:47:42.482] EVENT : config-sentinel runserver stopped/1 name="sbin/vespa-config-sentinel -c hosts/ip-172-31-83-99.ec2.internal (pid 592)" pid=592 exitcode=1 [2021-07-08 16:47:42.482] EVENT : config-sentinel runserver starting/1 name="sbin/vespa-config-sentinel -c hosts/ip-172-31-83-99.ec2.internal (pid 3687)" [2021-07-08 16:47:42.493] EVENT : config-sentinel sentinel.config-sentinel started/1 name="config-sentinel" [2021-07-08 16:48:02.221] EVENT : config-sentinel runserver stopping/1 name="sbin/vespa-config-sentinel -c hosts/ip-172-31-83-99.ec2.internal (pid 3687)" why="got signal 15" [2021-07-08 16:49:32.249] EVENT : config-sentinel runserver stopping/1 name="sbin/vespa-config-sentinel -c hosts/ip-172-31-83-99.ec2.internal (pid 3687)" why="got signal 15" [2021-07-08 16:49:32.249] EVENT : configproxy runserver stopping/1 name="java -Xms32M -Xmx256M -XX:CompressedClassSpaceSize=32m -XX:MaxDirectMemorySize=32m -XX:ThreadStackSize=256 -XX:MaxJavaStackTraceDepth=1000 -XX:-OmitStackTraceInFastThrow -XX:+ExitOnOutOfMemoryError -Dproxyconfigsources=tcp/ip-172-31-83-99.ec2.internal:19070 -Djava.io.tmpdir=/opt/vespa/tmp -XX:ActiveProcessorCount=2 -cp libexec/vespa/patches/configproxy:lib/jars/config-proxy-jar-with-dependencies.jar com.yahoo.vespa.config.proxy.ProxyServer 19090 (pid 17086)" why="got signal 15" [2021-07-08 16:49:32.252] EVENT : configproxy configproxy.com.yahoo.vespa.config.proxy.ProxyServer stopping/1 name=configproxy why="shutdown rpcServer" [2021-07-08 16:49:32.264] EVENT : configproxy configproxy.com.yahoo.vespa.config.proxy.ProxyServer stopping/1 name=configproxy why="cancel configClient" [2021-07-08 16:49:32.274] EVENT : configproxy configproxy.com.yahoo.vespa.config.proxy.ProxyServer stopping/1 name=configproxy why=flush [2021-07-08 16:49:32.274] EVENT : configproxy configproxy.com.yahoo.vespa.config.proxy.ProxyServer stopping/1 name=configproxy why="close fileDistribution" [2021-07-08 16:49:32.274] EVENT : configproxy configproxy.com.yahoo.vespa.config.proxy.ProxyServer stopping/1 name=configproxy why="stop complete" [2021-07-08 16:49:32.274] EVENT : configproxy configproxy.com.yahoo.vespa.config.proxy.ProxyServer stopping/1 name=configproxy why="clean shutdown" [2021-07-08 16:49:32.441] EVENT : configproxy runserver stopped/1 name="java -Xms32M -Xmx256M -XX:CompressedClassSpaceSize=32m -XX:MaxDirectMemorySize=32m -XX:ThreadStackSize=256 -XX:MaxJavaStackTraceDepth=1000 -XX:-OmitStackTraceInFastThrow -XX:+ExitOnOutOfMemoryError -Dproxyconfigsources=tcp/ip-172-31-83-99.ec2.internal:19070 -Djava.io.tmpdir=/opt/vespa/tmp -XX:ActiveProcessorCount=2 -cp libexec/vespa/patches/configproxy:lib/jars/config-proxy-jar-with-dependencies.jar com.yahoo.vespa.config.proxy.ProxyServer 19090 (pid 17086)" pid=17086 exitcode=0 [2021-07-08 16:50:42.497] EVENT : config-sentinel sentinel.config-sentinel stopping/1 name="config-sentinel" why="Timed out while subscribing to 'cloud.config.sentinel', configid 'hosts/ip-172-31-83-99.ec2.internal'" [2021-07-08 16:50:42.499] EVENT : config-sentinel runserver stopped/1 name="sbin/vespa-config-sentinel -c hosts/ip-172-31-83-99.ec2.internal (pid 3687)" pid=3687 exitcode=1

2021-07-08 16:46:42.169] WARNING : configserver Container.com.yahoo.container.di.Container Failed to set up first component graph due to error when constructing one of the components\nexception=\ncom.yahoo.container.di.componentgraph.core.ComponentNode$ComponentConstructorException: Error constructing 'com.yahoo.vespa.config.server.tenant.TenantRepository': null\nCaused by: java.lang.RuntimeException: Could not create all tenants when bootstrapping, failed to create: [default]\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.bootstrapTenants(TenantRepository.java:290)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.<init>(TenantRepository.java:210)\n [2021-07-08 16:46:42.169] WARNING : configserver Container.com.yahoo.jdisc.core.ApplicationLoader Exception thrown while activating application.\nexception=\ncom.yahoo.container.di.componentgraph.core.ComponentNode$ComponentConstructorException: Error constructing 'com.yahoo.vespa.config.server.tenant.TenantRepository': null\nCaused by: java.lang.RuntimeException: Could not create all tenants when bootstrapping, failed to create: [default]\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.bootstrapTenants(TenantRepository.java:290)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.<init>(TenantRepository.java:210)\n [2021-07-08 16:46:42.170] WARNING : configserver stderr com.yahoo.container.di.componentgraph.core.ComponentNode$ComponentConstructorException: Error constructing 'com.yahoo.vespa.config.server.tenant.TenantRepository': null [2021-07-08 16:46:42.170] WARNING : configserver stderr Caused by: java.lang.RuntimeException: Could not create all tenants when bootstrapping, failed to create: [default] [2021-07-08 16:46:42.170] WARNING : configserver stderr \tat com.yahoo.vespa.config.server.tenant.TenantRepository.bootstrapTenants(TenantRepository.java:290) [2021-07-08 16:46:42.170] WARNING : configserver stderr \tat com.yahoo.vespa.config.server.tenant.TenantRepository.<init>(TenantRepository.java:210) [2021-07-08 16:46:49.260] WARNING : configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler Timed out (timeout 25000) getting config name=sentinel,namespace=cloud.config,configId=hosts/ip-172-31-83-99.ec2.internal, will retry [2021-07-08 16:46:53.615] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:46:53.616] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:47:33 GMT [2021-07-08 16:47:10.211] WARNING : configserver Container.com.yahoo.vespa.config.server.modelfactory.ModelsBuilder Unexpected error building default.default\nexception=\njava.lang.RuntimeException: Document type 'stack' not found in application package\n\tat com.yahoo.vespa.model.content.cluster.SearchDefinitionBuilder.build(SearchDefinitionBuilder.java:25)\n\tat com.yahoo.vespa.model.content.cluster.ContentCluster$Builder.build(ContentCluster.java:114)\n\tat com.yahoo.vespa.model.content.Content$Builder.doBuild(Content.java:206)\n\tat com.yahoo.vespa.model.content.Content$Builder.doBuild(Content.java:190)\n\tat com.yahoo.config.model.builder.xml.ConfigModelBuilder.build(ConfigModelBuilder.java:73)\n\tat com.yahoo.config.model.ConfigModelRepo.buildModel(ConfigModelRepo.java:216)\n\tat com.yahoo.config.model.ConfigModelRepo.buildModels(ConfigModelRepo.java:202)\n\tat com.yahoo.config.model.ConfigModelRepo.readConfigModels(ConfigModelRepo.java:164)\n\tat com.yahoo.config.model.ConfigModelRepo.readConfigModels(ConfigModelRepo.java:90)\n\tat com.yahoo.vespa.model.VespaModel.<init>(VespaModel.java:199)\n\tat com.yahoo.vespa.model.VespaModel.<init>(VespaModel.java:172)\n\tat com.yahoo.vespa.model.VespaModelFactory.buildModel(VespaModelFactory.java:135)\n\tat com.yahoo.vespa.model.VespaModelFactory.createModel(VespaModelFactory.java:103)\n\tat com.yahoo.vespa.config.server.modelfactory.ActivatedModelsBuilder.buildModelVersion(ActivatedModelsBuilder.java:129)\n\tat com.yahoo.vespa.config.server.modelfactory.ActivatedModelsBuilder.buildModelVersion(ActivatedModelsBuilder.java:54)\n\tat com.yahoo.vespa.config.server.modelfactory.ModelsBuilder.buildModelVersions(ModelsBuilder.java:180)\n\tat com.yahoo.vespa.config.server.modelfactory.ModelsBuilder.buildModels(ModelsBuilder.java:120)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.loadApplication(SessionRepository.java:514)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.ensureApplicationLoaded(SessionRepository.java:453)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.loadSessionIfActive(SessionRepository.java:431)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.createRemoteSession(SessionRepository.java:332)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.addLocalSession(SessionRepository.java:198)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.createLocalSession(SessionRepository.java:726)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.createSessionFromId(SessionRepository.java:720)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.lambda$loadLocalSessions$2(SessionRepository.java:217)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n [2021-07-08 16:47:10.219] WARNING : configserver Container.com.yahoo.vespa.config.server.tenant.TenantRepository Failed to create tenant default\nexception=\njava.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not load local session 15\n\tat java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.bootstrapTenants(TenantRepository.java:280)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.<init>(TenantRepository.java:210)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.<init>(TenantRepository.java:141)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)\n\tat com.yahoo.container.di.componentgraph.core.ComponentNode.newInstance(ComponentNode.java:156)\n\tat com.yahoo.container.di.componentgraph.core.Node.constructInstance(Node.java:49)\n\tat java.base/java.util.HashMap$Values.forEach(HashMap.java:976)\n\tat com.yahoo.container.di.Container.constructComponents(Container.java:158)\n\tat com.yahoo.container.di.Container.getNewComponentGraph(Container.java:80)\n\tat com.yahoo.container.core.config.HandlersConfigurerDi.getNewComponentGraph(HandlersConfigurerDi.java:138)\n\tat com.yahoo.container.core.config.HandlersConfigurerDi.<init>(HandlersConfigurerDi.java:80)\n\tat com.yahoo.container.core.config.HandlersConfigurerDi.<init>(HandlersConfigurerDi.java:66)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.createConfigurer(ConfiguredApplication.java:336)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:141)\n\tat com.yahoo.container.standalone.StandaloneContainerApplication.start(StandaloneContainerApplication.java:157)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:44)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\nCaused by: java.lang.RuntimeException: Could not load local session 15\n\tat com.yahoo.vespa.config.server.session.SessionRepository.lambda$loadLocalSessions$4(SessionRepository.java:224)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1336)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.loadLocalSessions(SessionRepository.java:219)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.loadSessions(SessionRepository.java:181)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.loadSessions(SessionRepository.java:176)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.<init>(SessionRepository.java:167)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.createTenant(TenantRepository.java:344)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.bootstrapTenant(TenantRepository.java:304)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.lambda$bootstrapTenants$1(TenantRepository.java:273)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: java.util.concurrent.ExecutionException: com.yahoo.vespa.config.server.http.InternalServerException: Unexpected error building default.default\n\tat java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.lambda$loadLocalSessions$4(SessionRepository.java:221)\n\t... 13 more\nCaused by: com.yahoo.vespa.config.server.http.InternalServerException: Unexpected error building default.default\n\tat com.yahoo.vespa.config.server.modelfactory.ModelsBuilder.buildModels(ModelsBuilder.java:143)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.loadApplication(SessionRepository.java:514)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.ensureApplicationLoaded(SessionRepository.java:453)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.loadSessionIfActive(SessionRepository.java:431)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.createRemoteSession(SessionRepository.java:332)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.addLocalSession(SessionRepository.java:198)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.createLocalSession(SessionRepository.java:726)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.createSessionFromId(SessionRepository.java:720)\n\tat com.yahoo.vespa.config.server.session.SessionRepository.lambda$loadLocalSessions$2(SessionRepository.java:217)\n\t... 5 more\nCaused by: java.lang.RuntimeException: Document type 'stack' not found in application package\n\tat com.yahoo.vespa.model.content.cluster.SearchDefinitionBuilder.build(SearchDefinitionBuilder.java:25)\n\tat com.yahoo.vespa.model.content.cluster.ContentCluster$Builder.build(ContentCluster.java:114)\n\tat com.yahoo.vespa.model.content.Content$Builder.doBuild(Content.java:206)\n\tat com.yahoo.vespa.model.content.Content$Builder.doBuild(Content.java:190)\n\tat com.yahoo.config.model.builder.xml.ConfigModelBuilder.build(ConfigModelBuilder.java:73)\n\tat com.yahoo.config.model.ConfigModelRepo.buildModel(ConfigModelRepo.java:216)\n\tat com.yahoo.config.model.ConfigModelRepo.buildModels(ConfigModelRepo.java:202)\n\tat com.yahoo.config.model.ConfigModelRepo.readConfigModels(ConfigModelRepo.java:164)\n\tat com.yahoo.config.model.ConfigModelRepo.readConfigModels(ConfigModelRepo.java:90)\n\tat com.yahoo.vespa.model.VespaModel.<init>(VespaModel.java:199)\n\tat com.yahoo.vespa.model.VespaModel.<init>(VespaModel.java:172)\n\tat com.yahoo.vespa.model.VespaModelFactory.buildModel(VespaModelFactory.java:135)\n\tat com.yahoo.vespa.model.VespaModelFactory.createModel(VespaModelFactory.java:103)\n\tat com.yahoo.vespa.config.server.modelfactory.ActivatedModelsBuilder.buildModelVersion(ActivatedModelsBuilder.java:129)\n\tat com.yahoo.vespa.config.server.modelfactory.ActivatedModelsBuilder.buildModelVersion(ActivatedModelsBuilder.java:54)\n\tat com.yahoo.vespa.config.server.modelfactory.ModelsBuilder.buildModelVersions(ModelsBuilder.java:180)\n\tat com.yahoo.vespa.config.server.modelfactory.ModelsBuilder.buildModels(ModelsBuilder.java:120)\n\t... 13 more\n [2021-07-08 16:47:10.223] WARNING : configserver Container.com.yahoo.container.di.Container Failed to set up first component graph due to error when constructing one of the components\nexception=\ncom.yahoo.container.di.componentgraph.core.ComponentNode$ComponentConstructorException: Error constructing 'com.yahoo.vespa.config.server.tenant.TenantRepository': null\nCaused by: java.lang.RuntimeException: Could not create all tenants when bootstrapping, failed to create: [default]\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.bootstrapTenants(TenantRepository.java:290)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.<init>(TenantRepository.java:210)\n [2021-07-08 16:47:10.223] WARNING : configserver Container.com.yahoo.jdisc.core.ApplicationLoader Exception thrown while activating application.\nexception=\ncom.yahoo.container.di.componentgraph.core.ComponentNode$ComponentConstructorException: Error constructing 'com.yahoo.vespa.config.server.tenant.TenantRepository': null\nCaused by: java.lang.RuntimeException: Could not create all tenants when bootstrapping, failed to create: [default]\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.bootstrapTenants(TenantRepository.java:290)\n\tat com.yahoo.vespa.config.server.tenant.TenantRepository.<init>(TenantRepository.java:210)\n [2021-07-08 16:47:10.224] WARNING : configserver stderr com.yahoo.container.di.componentgraph.core.ComponentNode$ComponentConstructorException: Error constructing 'com.yahoo.vespa.config.server.tenant.TenantRepository': null [2021-07-08 16:47:10.224] WARNING : configserver stderr Caused by: java.lang.RuntimeException: Could not create all tenants when bootstrapping, failed to create: [default] [2021-07-08 16:47:10.224] WARNING : configserver stderr \tat com.yahoo.vespa.config.server.tenant.TenantRepository.bootstrapTenants(TenantRepository.java:290) [2021-07-08 16:47:10.224] WARNING : configserver stderr \tat com.yahoo.vespa.config.server.tenant.TenantRepository.<init>(TenantRepository.java:210) [2021-07-08 16:47:28.260] WARNING : configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler Timed out (timeout 25000) getting config name=sentinel,namespace=cloud.config,configId=hosts/ip-172-31-83-99.ec2.internal, will retry [2021-07-08 16:47:32.660] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:47:32.660] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:48:22 GMT [2021-07-08 16:47:42.480] WARNING : config-sentinel sentinel.config-sentinel Timeout getting config, please check your setup. Will exit and restart: Timed out while subscribing to 'cloud.config.sentinel', configid 'hosts/ip-172-31-83-99.ec2.internal' [2021-07-08 16:47:58.260] WARNING : configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler Timed out (timeout 15000) getting config name=sentinel,namespace=cloud.config,configId=hosts/ip-172-31-83-99.ec2.internal, will retry [2021-07-08 16:48:02.519] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:48:02.519] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:48:12 GMT [2021-07-08 16:48:34.260] WARNING : configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler Timed out (timeout 25000) getting config name=sentinel,namespace=cloud.config,configId=hosts/ip-172-31-83-99.ec2.internal, will retry [2021-07-08 16:48:38.555] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:48:38.555] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:48:58 GMT [2021-07-08 16:49:11.260] WARNING : configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler Timed out (timeout 25000) getting config name=sentinel,namespace=cloud.config,configId=hosts/ip-172-31-83-99.ec2.internal, will retry [2021-07-08 16:49:15.597] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:49:15.597] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:49:45 GMT [2021-07-08 16:49:32.254] WARNING : configproxy stderr com.yahoo.config.subscription.ConfigInterruptedException: java.lang.InterruptedException: sleep interrupted [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.config.subscription.impl.JRTConfigSubscription.subscribe(JRTConfigSubscription.java:150) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:133) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.config.subscription.impl.GenericConfigSubscriber.subscribe(GenericConfigSubscriber.java:48) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.vespa.config.proxy.Subscriber.subscribe(Subscriber.java:41) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.vespa.config.proxy.RpcConfigSourceClient.subscribeToConfig(RpcConfigSourceClient.java:162) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.vespa.config.proxy.RpcConfigSourceClient.getConfig(RpcConfigSourceClient.java:151) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.vespa.config.proxy.ProxyServer.resolveConfig(ProxyServer.java:81) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.vespa.config.proxy.ConfigProxyRpcServer.getConfigImpl(ConfigProxyRpcServer.java:281) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.vespa.config.proxy.ConfigProxyRpcServer.lambda$getConfigV3$1(ConfigProxyRpcServer.java:132) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat com.yahoo.vespa.config.proxy.ConfigProxyRpcServer.lambda$dispatchRpcRequest$14(ConfigProxyRpcServer.java:254) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [2021-07-08 16:49:32.254] WARNING : configproxy stderr \tat java.base/java.lang.Thread.run(Thread.java:834) [2021-07-08 16:49:32.255] WARNING : configproxy stderr Caused by: java.lang.InterruptedException: sleep interrupted [2021-07-08 16:49:32.255] WARNING : configproxy stderr \tat java.base/java.lang.Thread.sleep(Native Method) [2021-07-08 16:49:32.255] WARNING : configproxy stderr \tat com.yahoo.config.subscription.impl.JRTConfigSubscription.subscribe(JRTConfigSubscription.java:148) [2021-07-08 16:49:32.255] WARNING : configproxy stderr \t... 12 more [2021-07-08 16:49:32.262] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:50:32 GMT [2021-07-08 16:49:41.276] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:49:41.276] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:50:21 GMT [2021-07-08 16:49:51.300] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:49:51.300] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:50:41 GMT [2021-07-08 16:50:02.318] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:50:02.318] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:51:02 GMT [2021-07-08 16:50:14.339] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:50:14.339] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:51:14 GMT [2021-07-08 16:50:27.363] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:50:27.363] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:51:27 GMT [2021-07-08 16:50:41.382] WARNING : config-sentinel sentinel.config.frt.frtconnection Connection to tcp/localhost:19090 failed or timed out [2021-07-08 16:50:41.382] WARNING : config-sentinel sentinel.config.frt.frtconnection FRT Connection tcp/localhost:19090 suspended until 2021-07-08 16:51:41 GMT [2021-07-08 16:50:42.497] WARNING : config-sentinel sentinel.config-sentinel Timeout getting config, please check your setup. Will exit and restart: Timed out while subscribing to 'cloud.config.sentinel', configid 'hosts/ip-172-31-83-99.ec2.internal' [centos@ip-172-31-83-99 ~]$

bratseth commented 3 years ago

I see. That should be doable, although I haven't tried it.

sirganya commented 3 years ago

Thanks! That worked. I appreciate the help.

bratseth commented 3 years ago

Great to hear, closing.

sirganya commented 2 years ago

oops I did it again, I upgraded Vespa on a CentOs server, not using docker. The connection at tcp/localhost:19090 fails and I get this in the logs stderr

vespa-proton-bin: /builddir/build/BUILD/vespa-7.594.36/searchcore/src/vespa/searchcore/proton/server/documentdb.cpp:433: void proton::DocumentDB::applySubDBConfig(const proton::DocumentDBConfig&, proton::DocumentDB::SerialNum, const proton::ReconfigParams&): AssertionnewDocType != nullptr' failed.

kkraune commented 2 years ago

Just curious, if you use https://docs.vespa.ai/en/vespa-quick-start.html and deploy from the root of your app package instead, do you get an error then?

sirganya commented 2 years ago

I'm running a local version using docker and the application deploys successfully.

sirganya commented 2 years ago

The logs look ok up until this point

[2022-06-21 15:27:50] searchnode       Filedistributorrpc config is changed
[2022-06-21 15:27:50] searchnode       Add document database: doctypename(file), configid(stack/search/cluster.stack/file)
[2022-06-21 15:27:50] searchnode       Add document database: doctypename(filter), configid(stack/search/cluster.stack/filter)
[2022-06-21 15:27:50] searchnode       vespa-proton-bin: /builddir/build/BUILD/vespa-7.594.36/searchcore/src/vespa/searchcore/proton/server/documentdb.cpp:433: void proton::DocumentDB::applySubDBConfig(const proton::DocumentDBConfig&, proton::DocumentDB::SerialNum, const proton::ReconfigParams&): Assertion `newDocType != nullptr' failed.
[2022-06-21 15:27:51] config-sentinel  searchnode: incremented restart penalty to 62.000 seconds
[2022-06-21 15:27:51] config-sentinel  searchnode: incremented restart penalty to 126.000 seconds
[2022-06-21 15:27:51] config-sentinel  searchnode: will delay start by 124.604 seconds
[2022-06-21 15:27:57] container-clustercontroller Added node only event: Event: distributor.0: Altered node state in cluster state from 'U, t 1655824694' to 'U, t 1655825230, b 58'
[2022-06-21 15:27:57] container-clustercontroller Cluster event type SYSTEMSTATE @1655825277744: New cluster state version 11. Change from last: version: 10 => 11, storage: 0: description:  => Node not seen in slobrok., distributor: 0: [minUsedBits: 16 => 58, startTimestamp: 1655824694 => 1655825230]
[2022-06-21 15:27:57] container-clustercontroller Cluster 'stack': Storing new cluster state version in ZooKeeper: 11
[2022-06-21 15:27:57] container-clustercontroller Cluster 'stack': Master moratorium complete: timed out waiting for all nodes to report in
[2022-06-21 15:27:57] container-clustercontroller Cluster 'stack': Publishing cluster state version 11
[2022-06-21 15:27:57] distributor      Distributor node ready. Done initializing. Giving out of sequence metric event. Config id is stack/distributor/0
[2022-06-21 15:27:57] distributor      Transitioning from baseline state 'Down' to 'Up' (cluster state version 11)
[2022-06-21 15:27:57] container-clustercontroller Cluster event type SYSTEMSTATE @1655825277804: Reset 1 start timestamps as all available distributors have seen newest cluster state.