search

vespa-engine / vespa

AI + Data, online. https://vespa.ai
https://vespa.ai
Apache License 2.0
5.58k stars 586 forks source link

vespa-deploy prepare fails with internal timeouts #22291

Closed sunil924 closed 2 years ago

sunil924 commented 2 years ago

vespa-deploy prepare /tmp/application.zip command returns Http Status code 504.

Vespa logs shows follwing messages :

2022-04-26 15:43:33.555] INFO : jdisc/configserver configserver-flags BundleEvent STARTED [2022-04-26 15:44:09.497] INFO : configproxy configproxy.com.yahoo.vespa.config.JRTConnection Connecting to tcp/vespa-dev-0.vespa-dev-internal.myNS.svc.cluster.local:19070 [2022-04-26 15:44:14.923] INFO : configserver Container.com.yahoo.container.handler.threadpool.DefaultContainerThreadpool Threadpool 'default-pool': min=8, max=100, queue=0 [2022-04-26 15:44:30.241] INFO : configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient Subscribe for 'name=cloud.config.sentinel,configId=hosts/vespa-dev-0.vespa-dev-internal.myNS.svc.cluster.local,fd1703c6a260ae5f7300326a8cd2b53f' failed, closing subscriber [2022-04-26 15:44:30.473] INFO : configproxy configproxy.com.yahoo.vespa.config.JRTConnection Connecting to tcp/vespa-dev-0.vespa-dev-internal.myNS.svc.cluster.local:19070 [2022-04-26 15:44:36.468] INFO : configproxy configproxy.com.yahoo.vespa.config.JRTConnection Connecting to tcp/vespa-dev-0.vespa-dev-internal.myNS.svc.cluster.local:19070 [2022-04-26 15:44:38.440] WARNING : configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler Timed out (timeout 25000) getting config name=cloud.config.sentinel,configId=hosts/vespa-dev-0.vespa-dev-internal.myNS.svc.cluster.local, will retry

I can confirm that config server is up. I have verified it using curl command curl http://vespa-dev-0.vespa-dev-internal.myNS.svc.cluster.local:19071 Response - "handlers" : [ { "id" : "com.yahoo.vespa.config.server.http.v2.HttpGetConfigHandler", "class" : "com.yahoo.vespa.config.server.http.v2.HttpGetConfigHandler", "bundle" : "configserver:7.559.12", "serverBindings" : [ "http://*/config/v2/tenant/*/application//", "http://*/config/v2/tenant/*/application///", "http:///config/v2/tenant//application//environment//region//instance//", "http://*/config/v2/tenant/*/application/*/environment/*/region/*/instance///" ] }, { "id" : "com.yahoo.container.usability.BindingsOverviewHandler", "class" : "com.yahoo.container.usability.BindingsOverviewHandler", "bundle" : "container-disc:7.559.12", "serverBindings" : [ "http:///" ] },..........

This was a new clean deployment. Could you please help me in understanding the issue and resolving it.

jobergum commented 2 years ago

Have you tried increasing the timeout? Large application packages, with many schemas, might take longer than the default timeout, see https://docs.vespa.ai/en/reference/vespa-cmdline-tools.html#vespa-deploy

vespa-deploy -t 60 prepare
kkraune commented 2 years ago

closing - duplicate of #22315