Some dense retrieval models use different tokenization methods for queries and documents, such as TCT-ColBERT. To support these models, repconc detects whether the tokenizer.call has an argument named 'input_text_type'. If it has, the type will be set to 'query' or 'doc'. Therefore, the tokenizer can know whether the input is query or document and has customized behavior.
Some dense retrieval models use different tokenization methods for queries and documents, such as TCT-ColBERT. To support these models, repconc detects whether the tokenizer.call has an argument named 'input_text_type'. If it has, the type will be set to 'query' or 'doc'. Therefore, the tokenizer can know whether the input is query or document and has customized behavior.