Closed donhardman closed 1 month ago
@donhardman as discussed, pls update the example.
I do not see the issue here - boost correctly set for both terms then it get into count at sphinxsearch.cpp#L4620
The only issue I see that for non plain version fIDF
got 0.0
for a
term due to the formula at sphinxsearch.cpp#L4600
if ( !tQuery.m_bPlainIDF )
{
// bm25 variant, idf = log((N-n+1)/n), as per Robertson et al
//
// idf \in [-log(N), log(N)]
// weight \in [-NumWords*log(N), NumWords*log(N)]
// we want weight \in [0, 1] range
// we prescale idfs and get weight \in [-0.5, 0.5] range
// then add 0.5 as our final step
//
// for the record, idf = log((N-n+0.5)/(n+0.5)) in the original paper
// but our variant is a bit easier to compute, and has a better (symmetric) range
float fLogTotal = logf ( float ( 1+iTotalClamped ) );
fIDF = logf ( float ( iTotalClamped-iTermDocs+1 ) / float ( iTermDocs ) )
/ ( 2*fLogTotal );
with the values from the case
fLogTotal = 1.38629436
iTotalClamped = 3
iTermDocs= 2
fIDF = logf ( float ( 3-2+1 ) / float ( 2 ) ) / ( 2*1.38629436 )
fIDF = logf ( float ( 2 ) / float ( 2 ) ) / ( 2*1.38629436 )
fIDF = logf ( 1 ) / ( 2*1.38629436 )
fIDF = 0 / ( 2*1.38629436 )
fIDF = 0
but it seems due to the artificially small index not sure this exact case should be fixed as it could affect whole matching
I can confirm that when a table contains sufficient rows, we can observe changes in the weights. Closing the issue.
Bug Description:
In some instances, when we attempt to use IDF boosters, the weight isn't applied to certain phrases in the match query as expected. Here's a Minimal Reproducible Example (MRE) to demonstrate this issue:
Interestingly, when we set the option
idf='plain,tfidf_normalized'
, the weights for all the same queries in the MRE change and work as expected. This suggests there might be an issue with the default IDF score calculation.Manticore Search Version:
Latest dev version
Operating System Version:
Ubuntu Jammy
Have you tried the latest development version?
Yes
Internal Checklist:
To be completed by the assignee. Check off tasks that have been completed or are not applicable.