search

cohere-ai / cohere-finetune

A tool that facilitates easy, efficient and high-quality fine-tuning of Cohere's models
MIT License
54 stars 5 forks source link

Question about Hyperparameters #1

Open yongzx opened 2 weeks ago

yongzx commented 2 weeks ago

Thank you for this wonderful repo!

I have a quick question: are the hyperparameters listed in the README optimized for both finetuning on a specific task and finetuning to support a new language (i.e., language adaptation)?

youran-qi commented 2 weeks ago

Hi @yongzx, thank you and hope this repo is helpful in your use case!

The hyperparameters in the README are not optimized for a specific task or for language adaptation. We decided these hyperparameters according to experiments in some very generic use cases. Therefore, they are more like some hyperparameters you can start with, but you probably want to do some hyperparameter search to find the best hyperparameters for your specific task, e.g., language adaptation, etc.

Also, there are some validation codes to make sure the hyperparameters are in the "valid ranges" (see here). If you believe the best hyperparameters for your specific task are out of these "valid ranges", you can also use your out-of-range hyperparameters, but don't forget to remove / comment these validation codes.