search

terrierteam / pyterrier_colbert

83 stars 35 forks source link

The resulting query_token often contains noise #73

Open talk2much opened 4 months ago

talk2much commented 4 months ago

When using code to get PRF:

prf_rank = pytcolbert.prf(rerank=False,fb_docs=10,fb_embs=10)
df=prf_rank.search(query).head(1)
query_toks=df.iloc[0]['query_toks']

The resulting query_token often contains noise, such as: ['coating', 'smuggling', 'radioactive', 'kazakhstan', 'legislation', '##b', '12', 'minister', '"', '"'] query_token has unrelated symbols or numbers instead of words.

Is this due to the use of the robust2004 dataset? Is the noise caused by selecting the document during the PRF process and extracting the contents of the document set other than ?

In the indexing phase, I used the code from the pull request. That is, when using the robust2004 document set, the contents of line["text"] are extracted. Is this also required in the PRF phase?

cmacdonald commented 4 months ago

All PRF approaches can extract noisy expansion tokens. But you will find that their weight is very small (there is another query weight column).

cmacdonald commented 4 months ago

Our ColBERT PRF journal paper has examples of useful and non-useful tokens.