How does Vespa handle ColBERT type queries?

[jiangsiYang]

I practiced with the notebook from https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/mother-of-all-embedding-models-cloud.ipynb but I have some confusion here. This still uses dense_rep for matching, right? So, how should I use the colbert_rep field for matching? Thank you.

response: VespaQueryResponse = app.query(
yql="select id, text from m where userQuery() or ({targetHits:10}nearestNeighbor(dense_rep,q_dense))",
ranking="m3hybrid",
query=query[0],
body={**query_fields},
)

Querying this way results in an error.

response: VespaQueryResponse = app.query(
yql="select act_id, text from act_tag where ({targetHits:10}nearestNeighbor(colbert_rep,q_colbert))",
ranking="m3hybrid",
query=query[0],
body={**query_fields},
)

error:

VespaError: [{'code': 3, 'summary': 'Illegal query', 'source': 'qjl_content', 'message': 'NEAREST_NEIGHBOR {field=colbert_rep,queryTensorName=q_colbert,hnsw.exploreAdditionalHits=0,distanceThreshold=Infinity,approximate=true,targetHits=10} tensor query(q_colbert) must have exactly 1, indexed dimension, but was: tensor(qt{},x[1024])'}]

[jobergum]

As the error demonstrates, the nearestNeighbor operator only accepts a single dense vector representation and not a mixed tensor (multi-vector). If you really want to retrieve with the multi vector representation, you can look at this example https://pyvespa.readthedocs.io/en/latest/examples/colbert_standalone_Vespa-cloud.html

• In essence you need multiple nearestNeighbor operator in the same vespa query request and all the tensors must be defined in the schema.