search

OpenBMB / UltraFeedback

A large-scale, fine-grained, diverse preference dataset (and models).
MIT License
297 stars 16 forks source link

Training details for reproducing UltraCM #9

Open huchinlp opened 9 months ago

huchinlp commented 9 months ago

Thank you so much for sharing the data. It's very helpful for the RLHF community!

I found some hyper-parameters for training UltraCM in your paper, but I am also confused by the following questions:

  1. How do you prepare the training examples? It seems that the instruction, completion, the feedback, and the overall score are filled into the ultracm_instruction_template as defined in your demo page. But I'm not sure...
  2. How is the loss calculated? Did you apply masking to the input content, including the instruction and completion?
  3. Did you compare tuning a critique model from an SFT model versus a pretrained checkpoint?

Thanks again for your efforts!