TREC 2024 - Workflow Record

[Yuv-sue1005]

This issue isn't a problem to be fixed, rather it is a record to keep track of undergraduate student's work on TREC 2024. In a series of comments, contributors can write about their work (at a high level) on specific tracks. This issue will be regularly updated upon finishing certain tasks/tracks.

[Yuv-sue1005]

Here's my contributions to TREC's NeuCLIR track.

• Task: Cross-Language Technical Documents
  • Assisted in creation of a f-stage, gte-qwen2 dense retrieval baseline. Assisted in writing scripts for encoding corpus and queries, and then running retrieval with aforementioned embeddings.
  • Created BM25 document translation and query translation baselines. Indexed corpus, ran baselines, and evaluated all created runs.
  • Created a SPLADE document translated baseline. Encoded the corpus and queries using splade model, indexed corpus embeddings, ran the baseline, and evaluated results. This baseline was not used in submissions, given it's surprisingly low eval scores.
  • Created and ran several data munging scripts to reformat queries and corpus.
• Task: Multilingual Retrieval (MLIR)
  • Created and used multiple reformatting scripts, the most notable one being converting a trec run into retrieval results format.
  • Created SPLADE, BM25-dt, BM25-qt, and PLAID baselines for task (basically all f-stage baselines). Ran all baselines, evaluated runs, RRFed runs into fusion runs, and then evaluated fusion runs. Fusion runs were sent off to mono stage.
  • Fused (with RRF) and evaluated post-mono and post-listo runs.
• Task: Cross-language Retrieval (CLIR)
  • Taking post-mono fused runs from MLIR, I combined the zho, rus, and fas runs to create a top 300 retrieval results run for CLIR. This new retrieval results file was then sent off to list-wise reranking.
• Task: Cross-Language Report Generation
  • Created SPLADE, BM25-dt, and PLAID baselines for task (all f-stage baselines). Ran all baselines, evaluated runs, RRFed runs into fusion runs, and then evaluated fusion runs. Fusion runs were sent off to mono stage.
  • Created and used multiple reformatting scripts.

[Stefan824]

Cross-Language Technical Document Tasks

1. Data Preprocessing

• Preprocessed all relevant data to a Pyserini-compatible format (corpus, topics/queries, and qrels).

2. Dense Embedding Experiments Reproduction

• Reproduced existing experiments using Pyserini for pre-encoded corpus and queries.
• Gained insights into how Pyserini handles external models.

3. Dense Retrieval Baseline (GTE-Qwen2)

• Developed and implemented the dense retrieval baseline using the GTE-Qwen2 model (a dense embedding model not natively supported by Pyserini).
• Scripted the encoding process for both corpus and queries, converting them into a Pyserini-compatible format.
• Executed indexing, searching, and evaluation using Pyserini.

4. Integration for Pyserini with the Qwen Model

• Added connector code to integrate the GTE-Qwen2 dense embedding model with Pyserini.
• Currently testing model performance; results are pending.

5. BM25 and SPLADE Baselines (Document and Query Translation)

• Assisted with setting up baselines using supported models from Pyserini/Anserini.

6. Runs Fusion

• Scripted processes for fusing outputs from different baselines.
• Finalized outstanding runs with run fusion strategies.

---

Multi-Language Information Retrieval (MLIR) and Cross-Language Information Retrieval (CLIR) Tasks

1. Baseline Setup and Reproduction

• Located and reproduced baselines from previous years.
• Contributed to the setup of all first-stage retrieval baselines.

---

Report Generation Tasks

1. Report Request Handling

• Scripted tools to extract and format key information from report requests.

2. Prompt Engineering for GPT-4

• Engineered prompts for GPT-4 to break down report requests into sub-questions.
• Implemented scripts to generate results from these sub-questions.

3. First-Stage Retrieval for Reports

• Performed initial retrieval on report requests paired with corresponding sub-questions.

4. Reranking with Cohere Reranker

• Applied reranking using the Cohere reranker on the initial retrieval results.

[Yuv-sue1005]

Here's my contributions to the TREC RAG track.

• Implemented bge-reranker-v2 models to rank_llm as a pointwise reranker (like monoT5) for use in RAG. Implementation involved understanding, borrowing, modifying, and adding to pre-existing rank_llm code to work with the bge-reranker-v2 line of models. Now, anyone can use implemented bge models for pointwise reranking. The exact models implemented are BAAI/bge-reranker-base, BAAI/bge-reranker-large, BAAI/bge-reranker-v2-m3, BAAI/bge-reranker-v2-gemma, and BAAI/bge-reranker-v2-minicpm-layerwise.