parkjunsoo91
commented
1 month ago Hi, thank you for your interest in our work.
The paper's description for Base-data is indeed little, because we thought that the specifics of the Base-data are too subtle and not the focus of the paper. Unfortunately it is currently difficult to share the exact processing code, nor the training code, as it involves libraries used internally in the company. Also I'm afraid we cannot share the exact data because of license issue. However I'll be happy to clarify the process for building Base-data.
For PKU-SafeRLHF, we used RLHFlow/PKU-SafeRLHF-30K-standard dataset.
For the single-scoring data, we have Helpsteer and Ultrafeedback:
The Helpsteer data is processed as follows:
- for each instance, randomly sample 1~5 criteria from the 5 existing criteria to be evaluated on simultaneously.
- use the prompt from the original HelpSteer paper (https://arxiv.org/abs/2311.09528). This will become very long if all 5 criteria are selected.
The Ultrafeedback data was processed as follows:
- exclude Flan subset.
- for each of the 43K instructions, for each of the 4 criteria, create one evaluation instance.
- for every evaluation instance, randomly sample 1~4 responses from the original data for simultaneous scoring.
- exclude "N/A" scores
- use the prompts from the Ultrafeedback github repo (https://github.com/OpenBMB/UltraFeedback/blob/main/src/data_annotation/preference_templates.py)
For preparing 3.1k pairwise data from Ultrafeedback, we do the following:
- from the single-scoring Ultrafeedback instances that satisfy the following condition:
- is evaluated on "instruction-following" aspect
- has 2 candidate responses
- the 2 scores are different
- the higher scoring sample has high quality, with overall score >= 8.0 and annotation rating average >= 4.75
- randomly select 40% of such instances and convert them to pairwise preference data, and exclude them from the single scoring UltraFeedback data.
- for the prompt, take the Ultrafeedback instruction-following prompt's instruction+criteria and fuse it into LLMBar prompt to make a new prompt. We simply added a (4) in LLMBar prompt and copy-pasted the criteria.
Hope this helps!
Hi,
Thank you for your excellent work. I am very interested in the paper and am currently trying to reproduce your results.
The paper mentions that Base-data includes a subset of PKU-SafeRLHF, but it doesn't provide details on the selection criteria for this subset. I also noticed that you converted 3.1k scoring data points from Ultrafeedback into a pairwise preference format.
Could you please provide the scripts to generate Base-data? Alternatively, would it be possible to share the Base-data itself?
Furthermore, I am curious about the method you used to train the model using single scoring data. The paper does not provide a prompt template for this process. Could you please explain how you trained with single scoring data?
I would really appreciate it if you could provide the training code. It will be of great help to me.
Best regards.