[Bug]: Eval Dataset passed as None when trying to Finetune Embeddings Model

Bug Description

On llama-index 0.11.22 and llama-index-finetuning 0.2.1. I was attempting to follow the documentation to finetune the BAAI/bge-small-en-v1.5 model on my own dataset. I attempted two different approaches that were present in the llama-index documentation: Approach 1:

train_dataset = EmbeddingQAFinetuneDataset.from_json("train_dataset.json")
val_dataset = EmbeddingQAFinetuneDataset.from_json("val_dataset.json")
finetune_engine = SentenceTransformersFinetuneEngine(
    train_dataset,
    model_id="BAAI/bge-small-en-v1.5",
    model_output_path="arxiv_finetuned_model",
    val_dataset=val_dataset,
)
finetune_engine.finetune()
embed_model = finetune_engine.get_finetuned_model()
print(embed_model)
embed_model.save("finetuned_embedder")

Approach 2:

model_id = "BAAI/bge-small-en-v1.5"
model = SentenceTransformer(model_id)
with open("train_dataset.json", 'r+') as f:
    train_dataset = json.load(f)

with open("val_dataset.json", 'r+') as f:
    val_dataset = json.load(f)

dataset = train_dataset

corpus = dataset['corpus']
queries = dataset['queries']
relevant_docs = dataset['relevant_docs']

examples = []
for query_id, query in queries.items():
    node_id = relevant_docs[query_id][0]
    text = corpus[node_id]
    example = InputExample(texts=[query, text])
    examples.append(example)
loader = DataLoader(
    examples, batch_size=100
)
loss = losses.MultipleNegativesRankingLoss(model)
dataset = val_dataset

corpus = dataset['corpus']
queries = dataset['queries']
relevant_docs = dataset['relevant_docs']

evaluator = InformationRetrievalEvaluator(queries, corpus, relevant_docs)
warmup_steps = int(len(loader) * 15 * 0.1)

model.fit(
    train_objectives=[(loader, loss)],
    epochs=15,
    warmup_steps=warmup_steps,
    output_path='exp_finetune',
    show_progress_bar=True,
    evaluator=evaluator, 
    evaluation_steps=50,
)

However in both cases, I get the following error ValueError: You have setargs.eval_strategyto steps but you didn't pass aneval_datasettoTrainer. Either setargs.eval_strategytonoor pass aneval_dataset.

Looking at the source code I see that in Sentence-Transformers fit-mixin eval dataset is passed in as None which may cause the issue. I double checked the format of my train and val jsons, and those are correct. I do want to perform the finetuning with validation, so any help would be much appreciated!

Version

0.11.22

Steps to Reproduce

Run either of the above approaches and the error should occur. Also attempting to run the code present in the documentation here and here should result in the same error.

Relevant Logs/Tracbacks

Traceback (most recent call last):
  File "/storage/ice1/6/XXX/finetune.py", line 53, in <module>
    finetune_engine.finetune()
  File "/home/hice1/XXX/.conda/envs/new_llm/lib/python3.10/site-packages/llama_index/finetuning/embeddings/sentence_transformer.py", line 82, in finetune
    self.model.fit(
  File "/home/hice1/XXX/.conda/envs/new_llm/lib/python3.10/site-packages/sentence_transformers/fit_mixin.py", line 356, in fit
    trainer = SentenceTransformerTrainer(
  File "/home/hice1/XXX/.conda/envs/new_llm/lib/python3.10/site-packages/sentence_transformers/trainer.py", line 205, in __init__
    super().__init__(
  File "/home/hice1/XXX/.conda/envs/new_llm/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 165, in wrapped_func
    return func(*args, **kwargs)
  File "/home/hice1/XXX/.conda/envs/new_llm/lib/python3.10/site-packages/transformers/trainer.py", line 419, in __init__
    raise ValueError(
ValueError: You have set `args.eval_strategy` to steps but you didn't pass an `eval_dataset` to `Trainer`. Either set `args.eval_strategy` to `no` or pass an `eval_dataset`.

Hey @mjohal3! I'm here to help you with any bugs, questions, or contributions you have. Let's work through this together!

To resolve the ValueError related to args.eval_strategy when the evaluation dataset is passed as None, ensure that the evaluator is set to None when no validation dataset is provided. The SentenceTransformersFinetuneEngine class handles this by setting evaluator to None if val_dataset is None. Make sure you are not passing any evaluation-related arguments that require an evaluator when val_dataset is None.

Here's the relevant part of the code that handles this:

evaluator: Optional[InformationRetrievalEvaluator] = None
if val_dataset is not None:
    evaluator = InformationRetrievalEvaluator(
        val_dataset.queries, val_dataset.corpus, val_dataset.relevant_docs
    )
self.evaluator = evaluator

When calling the finetune method, do not pass parameters that depend on the evaluator being present if val_dataset is None. The finetune method will handle the absence of an evaluator correctly as long as no evaluation steps are required ^[1].

To continue talking to Dosu, mention @dosu.

run-llama / llama_index