Why no eval dataset

[WYY220062]

Hi! I have a basic question, why the whole fine rune does need the eval dataset?

[2U1]

There is no need for the eval dataset. If you want use it, you could. However it's not necessary.

[WYY220062]

Hi! I am wondering if I can add eval dataset part. Can you point me any sample code or reference?

[2U1]

You could change the function in make_supervised_data_module.

https://github.com/2U1/Phi3-Vision-ft/blob/2eddf0df610ef93e214092177294dec4dba027f3/src/training/data.py#L166-L175

[WYY220062]

Do you mean I can make simple change like this:

def make_supervised_data_module(processor, data_args):
    """Make dataset and collator for supervised fine-tuning."""
    sft_dataset = LazySupervisedDataset(
        data_path=data_args.data_path, processor=processor, data_args=data_args
    )
    eval_dataset = LazySupervisedDataset(
        data_path=/evaluation data path, processor=processor, data_args=data_args
    )
    data_collator = DataCollatorForSupervisedDataset(tokenizer=processor.tokenizer)

    return dict(train_dataset=sft_dataset,
                eval_dataset=eval_dataset,
                data_collator=data_collator)

I did a simple try, let the eval_dataset = sft_dataset. But I didnt see eval dataset play any role in the training. Do I miss anything? My fine tune job seems have overfitting problem. I want to have eval loss on the wandb dashboard too. May I have any suggestion?

[2U1]

Yes, that is just the part for making the eval dataset. You should add some parameters and metrics for the trainer. The metrics should be different what task you want. You can check the official phi3-vision finetuning code for it.

https://github.com/microsoft/Phi-3CookBook/blob/b3388e7780a5a7dc9988c91f1d46f1a606077141/code/04.Finetuning/vision_finetuning/finetune_hf_trainer_docvqa.py#L265-L322

[2U1]

Closing the issu for no updates.