Closed b41sh closed 4 weeks ago
Improved Test Variables in pruning.rs
The test variables in pruning.rs
were renamed from being prefixed with _
to their actual names (e13, e14, e15). This makes the code clearer and easier to understand.
Activated Test Assertions
The test assertions for the variables e13
, e14
, and e15
were activated to ensure they function as expected.
Public Access Granted to TermReader
in lib.rs
The TermReader
was added to the pub use
list in lib.rs
, meaning we've made it publicly accessible and ready for use.
Added New Dependency tantivy-common
The tantivy-common
dependency was added to our project. This new software element enhances our project's functionality.
Enhanced Field Normalization Handling
By using FieldNormReader
from tantivy
in inverted_index_reader.rs
, the handling of field normalization has been greatly improved.
Upgraded Term ID Tracking
We've introduced field_term_ids
. This new feature allows for better tracking of term ids for each field during term info collection.
Improved Index Slice Logic We've updated the way the index slice is created. It now handles score calculations and introduces new structures for managing field norms and total number tokens.
Optimized Document Collection Process
The DocIdsCollector
instantiation now uses TermReader
, which improves performance and clarity. Moreover, we've adjusted the scoring calculations, allowing for better score management.
Enhanced Field Norms Reading
In inverted_index_writer.rs
, the reading of field norms has been streamlined, ensuring correct extraction from the segment file. This will make our data more consistent and reliable.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
continue #16589
Tests
Type of change
This change isโ