search

facebookresearch / rlfh-gen-div

This is code for most of the experiments in the paper Understanding the Effects of RLHF on LLM Generalisation and Diversity
Other
32 stars 3 forks source link

How to integrate other hf models in the repo? #4

Open fmxFranky opened 4 months ago

fmxFranky commented 4 months ago

Hello, I really appreciate the fantastic work on this repository! It's been incredibly helpful for my projects. I have a quick question regarding the integration of additional hf models like microsoft/phi and google/gemma in this repo. Could you please provide some guidance or point me to the right resources on how to incorporate these models into this codebase smoothly?

RobertKirk commented 3 months ago

Hi, thanks for your interest in the repository. I think integrating new HF models would be doable. Most of the changes would happen in https://github.com/facebookresearch/rlfh-gen-div/blob/main/rlvsil/models/model_creation.py - you might need to adjust the details of the init function to make sure base_model_prefix and base_model_layers are set correctly. Probably the easiest way to start is to get the environment set up and installed, try running the training script with one of those model types and fixing the bugs that come up.

One issue might be that those newer HF models require a newer version of transformers than this codebase uses, and I think upgrading will cause incompatability with how this code does model parallelism and other things. Updating the code to work with newer transformers versions will likely be more difficult, and I don't know the kinds of changes you'd need to make there.

If you encounter specific issues then feel free to post them here and I'll try and help fix them!