search

vespa-engine / pyvespa

Python API for https://vespa.ai, the open big data serving engine
https://pyvespa.readthedocs.io/
Apache License 2.0
103 stars 32 forks source link

Deploy the Vespa application with docker for bge-m3 #778

Closed xjDUAN184 closed 6 months ago

xjDUAN184 commented 6 months ago

I have two container: A container and vespa container (two container all in one server) my python code(in A container): `from vespa.io import VespaResponse, VespaQueryResponse from FlagEmbedding import BGEM3FlagModel from vespa.package import Schema, Document, Field, FieldSet from vespa.package import RankProfile, Function, FirstPhaseRanking, ApplicationPackage from vespa.deployment import VespaDocker from vespa.application import Vespa

origin_model_path = '/nfs1/56dev/root/.cache/huggingface/hub/models--BAAI--bge-m3/snapshots/babcf60cae0a1f438d7ade582983d4ba462303c2' model = BGEM3FlagModel(origin_model_path, use_fp16=True) passage = [ "BGE M3 is an embedding model supporting dense retrieval, lexical matching and multi-vector interaction." ] passage_embeddings = model.encode( passage, return_dense=True, return_sparse=True, return_colbert_vecs=True ) m_schema = Schema( name="m", document=Document( fields=[ Field(name="id", type="string", indexing=["summary"]), Field( name="text", type="string", indexing=["summary", "index"], index="enable-bm25", ), Field( name="lexical_rep", type="tensor(t{})", indexing=["summary", "attribute"], ), Field( name="dense_rep", type="tensor(x[1024])", indexing=["summary", "attribute"], attribute=["distance-metric: angular"], ), Field( name="colbert_rep", type="tensor(t{}, x[1024])", indexing=["summary", "attribute"], ), ], ), fieldsets=[FieldSet(name="default", fields=["text"])], ) vespa_app_name = "m" vespa_application_package = ApplicationPackage( name=vespa_app_name, schema=[m_schema])

有关排序的配置文件

semantic = RankProfile( name="m3hybrid", inputs=[ ("query(q_dense)", "tensor(x[1024])"), ("query(q_lexical)", "tensor(t{})"), ("query(q_colbert)", "tensor(qt{}, x[1024])"), ("query(q_len_colbert)", "float"), ], functions=[ Function( name="dense", expression="cosine_similarity(query(q_dense), attribute(dense_rep),x)", ), Function( name="lexical", expression="sum(query(q_lexical) attribute(lexical_rep))" ), Function( name="max_sim", expression="sum(reduce(sum(query(q_colbert) attribute(colbert_rep) , x),max, t),qt)/query(q_len_colbert)", ), ], first_phase=FirstPhaseRanking( expression="0.4dense + 0.2lexical + 0.4*max_sim", rank_score_drop_limit=0.0 ), match_features=["dense", "lexical", "max_sim", "bm25(text)"], )

match_features 指的是在每次搜索结果中包含的排名特征列表。

m_schema.add_rank_profile(semantic)

app = Vespa(url="http://192.168.2.53", port=19071, application_package=vespa_application_package)

vespa_docker = VespaDocker(port=8080)

app = vespa_docker.deploy(application_package=vespa_application_package)

vespa_docker

vespa_fields1 = { "text": passage[0], "lexical_rep": { key: float(value) for key, value in passage_embeddings["lexical_weights"][0].items() }, "dense_rep": passage_embeddings["dense_vecs"][0].tolist(), "colbert_rep": { index: passage_embeddings["colbert_vecs"][0][index].tolist() for index in range(passage_embeddings["colbert_vecs"][0].shape[0]) }, }

vespa_fields = { "id": "0", "text": passage[0], "fields": { "lexical_rep": { key: float(value) for key, value in passage_embeddings["lexical_weights"][0].items() }, "dense_rep": passage_embeddings["dense_vecs"][0].tolist(), "colbert_rep": { index: passage_embeddings["colbert_vecs"][0][index].tolist() for index in range(passage_embeddings["colbert_vecs"][0].shape[0]) }, } }

def callback(response: VespaResponse, id: str): if not response.is_successful(): print(f"Error when feeding document {id}: {response.get_json()}") else: print(f"sucess feed")

app.feed_data_point(fields=vespa_fields, schema="m",

data_id=0, callback=callback)

app.feed_iterable([vespa_fields], schema="m", callback=callback) ` When I build the application this way： app = Vespa(url="http://192.168.2.53", port=19071, application_package=vespa_application_package) bug info: dict_keys(['dense_vecs', 'lexical_weights', 'colbert_vecs']) Error when feeding document 0: {'Exception': '503 Server Error: Service Unavailable for url: http://127.0.0.1:19071/document/v1/m/m/docid/0', 'id': '0', 'message': 'Exception during feed_data_point'}

When I build the application this way： vespa_docker = VespaDocker() app = vespa_docker.deploy(application_package=vespa_application_package) bug info: ----------using 2*GPUs---------- dict_keys(['dense_vecs', 'lexical_weights', 'colbert_vecs']) Traceback (most recent call last): File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/urllib3/connectionpool.py", line 496, in _make_request conn.request( File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/urllib3/connection.py", line 400, in request self.endheaders() File "/root/anaconda3/envs/uniem/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/root/anaconda3/envs/uniem/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/root/anaconda3/envs/uniem/lib/python3.10/http/client.py", line 976, in send self.connect() File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/transport/unixconn.py", line 27, in connect sock.connect(self.unix_socket) FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment( File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment raise reraise(type(error), error, _stacktrace) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise raise value.with_traceback(tb) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/urllib3/connectionpool.py", line 496, in _make_request conn.request( File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/urllib3/connection.py", line 400, in request self.endheaders() File "/root/anaconda3/envs/uniem/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/root/anaconda3/envs/uniem/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/root/anaconda3/envs/uniem/lib/python3.10/http/client.py", line 976, in send self.connect() File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/transport/unixconn.py", line 27, in connect sock.connect(self.unix_socket) urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/api/client.py", line 213, in _retrieve_server_version return self.version(api_version=False)["ApiVersion"] File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version return self._result(self._get(url), json=True) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/utils/decorators.py", line 44, in inner return f(self, *args, kwargs) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/api/client.py", line 236, in _get return self.get(url, self._set_request_timeout(kwargs)) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/requests/adapters.py", line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/root/anaconda3/envs/uniem/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/root/anaconda3/envs/uniem/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/root/.vscode-server/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in cli.main() File "/root/.vscode-server/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main run() File "/root/.vscode-server/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file runpy.run_path(target, run_name="main") File "/root/.vscode-server/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path return _run_module_code(code, init_globals, run_name, File "/root/.vscode-server/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code _run_code(code, mod_globals, init_globals, File "/root/.vscode-server/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code exec(code, run_globals) File "/nfs1/56dev/root/dxj/AI问答/embedding微调/FlagEmbedding/vespa_test.py", line 92, in app = vespa_docker.deploy(application_package=vespa_application_package) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/vespa/deployment.py", line 167, in deploy return self._deploy_data(application_package, application_package.to_zip(), debug) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/vespa/deployment.py", line 283, in _deploy_data self._run_vespa_engine_container( File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/vespa/deployment.py", line 320, in _run_vespa_engine_container client = docker.from_env(timeout=DOCKER_TIMEOUT) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/client.py", line 94, in from_env return cls( File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/client.py", line 45, in init self.api = APIClient(*args, **kwargs) File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/api/client.py", line 197, in init self._version = self._retrieve_server_version() File "/root/anaconda3/envs/uniem/lib/python3.10/site-packages/docker/api/client.py", line 220, in _retrieve_server_version raise DockerException( docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

besides： My docker container uses docker run -itd --name vespa -v /nfs1/:/nfs1/ --network host vespaengine/vespa:latest

or use docker-compose.yaml

Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.

Docker Compose file to start up a sample multinode Vespa system

The start sequence is:

- Start the 3 nodes 0,1,2 with a configserver and wait for successful start using healthcheck on a configserver.

- Start the rest of the nodes.

- After the 10 nodes are started, deploy the Vespa application package.

Notes:

- The settings for the healthcheck are just to bail out fast in case of config server start problems,

and not tested for production use.

- See https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA#readme

for background for the full configuration.

- The port mappings are for observability and status pages when all containers run on the same host,

mapping to a unique host port.

In a true multinode system, this can be simplified by mapping to the same port.

Normally, only 19071 (deployments) and 8080 (feed/query) needs to be mapped out of the Docker network.

services: node0: container_name: node0 hostname: node0.vespanet image: vespaengine/vespa networks:

networks: vespanet: driver: bridge name: vespanet

What are the problems when I use docker to deploy vespa?

thomasht86 commented 6 months ago

From what I can decode, it seems like you are trying to deploy from a container. The issue seem to be that the container does not have access to a running Docker Engine. If you want to enable this, you need to look at something like this: https://kodekloud.com/blog/run-docker-in-docker-container/ Closing the issue as this is not pyvespa-related.