Is using SFT data directly as calibration data the best option? Does it cause performance fluctuations when I have more (e.g. 28w) or less (e.g. 1k) fine-tuned data? Also, would using datasets from other domains as calibration datasets cause a gain or huge loss in performance? Thanks for sharing your practical experience.
Hi, I have recently also been considering using my own calibration dataset. Could you provide any guidance or tutorials on how to use a custom calibration dataset? Thank you in advance for your kind cooperation.
Is using SFT data directly as calibration data the best option? Does it cause performance fluctuations when I have more (e.g. 28w) or less (e.g. 1k) fine-tuned data? Also, would using datasets from other domains as calibration datasets cause a gain or huge loss in performance? Thanks for sharing your practical experience.