Open monchopena opened 4 weeks ago
I am encountering the same Vespa issue when trying to deploy Danswer, any suggestions would be appreciated.
Same issue here
I found the issue. Vespa k8s service labels don't match the labels applied to the Vespa pod by Danswer here. For now the fix is to add the following to your Danswer Helm chart values:
vespa:
podLabels:
app: vespa
app.kubernetes.io/instance: danswer
but a more permanent fix would be to make the service labels take their value from {{ .Values.podLabels }}
here.
I changed the podLabels:
$ k describe po vespa-0
Name: vespa-0
Namespace: danswer
Priority: 0
Node: kontabo-control-plane/172.18.0.2
Start Time: Fri, 07 Jun 2024 18:48:43 +0200
Labels: app=vespa
app.kubernetes.io/instance=danswer
....
But the problem persists:
[2024-06-07 17:37:38.955] INFO configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler Timed out (timeout 25000) getting config name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local, will retry
[2024-06-07 17:37:45.377] INFO config-sentinel sentinel.config.frt.frtconfigagent No response / error from config server. This is normal before an application package is deployed. (key: name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local) (errcode=103, validresponse:0), trying again in 7.000000 seconds
[2024-06-07 17:37:57.287] INFO configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient Subscribe for 'name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local,aa9115ebf63cfc6721f75ada21d7cdfa' failed, closing subscriber: Subscribe for 'name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local' timed out (timeout was 44000 ms): name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local, Current generation: 0, Generation changed: false, Config changed: false
[2024-06-07 17:37:57.291] INFO configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester Request failed: Failed request (No application exists) from Connection { Socket[addr=/10.244.0.182,port=59596,localport=19070] }\nConnection spec: tcp/vespa-0.vespa.danswer.svc.cluster.local:19070
---
I thinks this connection problem is not related with the podLabels.
Same here. I've tried swapping out multiple image tags as well, but I always get the same result.
Did you restart the Danswer pods after correcting the Vespa pod labels? Based on my reading of the various Danswer pod logs, one of the things the Danswer API server does on startup is connect to the Vespa service to set up the 'Application Package' (which your Vespa logs say is missing).
I restarted the Danswer pods and now is running properly! Thank you very much for your answer.
Hi 👋,
This is the log:
I tried with serveral configuration for
VESPA_CONFIGSERVERS=vespa-0.vespa.danswer.svc.cluster.local
But I couldn't start to work with Vespa in K8s.