[x] Refactoring the output format of the dataset to support flexible training & evaluation.
Main change: from content to old_content, subject and new_content.
Motivation: we should know the input length when calculating the loss over the target, this is necessary for GPT-like model fine-tuning.
[x] Add hyper-parameter information to the dataset to make it clear for each dataset.
Main change: write into a file to record the hyper-parameter for different datasets.
Motivation: this is to double-check the hyper-parameters of different filtering strategies, and then figure out the best one with the experimental results.
Training
[x] Adding the argument which enables the user to remove the loss in the input buggy code & commit message.
Conclusion: this kind of modification will reduce the loss value on train and dev by a large margin, which indicates that the previous wired train loss curve may be caused by the improper loss computation over inputs also. (Still verifying).
[x] Add the support for merging examples from the subset of FLAN.
[ ] Add support for evaluation on different benchmarks during pre-training after every X steps.
[x] Add the option to enable computing loss on instruction + target but not in the buggy code.
[x] Given the old_content and commit_message , just output the diff instead of the while file. It may make the evaluation harder since we should apply these diffs into the original content to obtain the new_content. But it has an advantage on that the old_content can be larger.
Progress: the current diff is obtained via a string-level comparison. May be worthy to try using line-level comparison.
[ ] Their experiments show that truncation on the file content (e.g., cutting the old_content to some relevant lines to make it shorter) will result in worse performance. Our current implementation mixes these two modes: in 20% time, the model will accept the whole file content as the input; in 80% time, the model will randomly cut the input file content into a reasonable range. Maybe 100% whole file content will be better?
[x] Experiment if adding filename is better (In Carper they do <NME> filename <....).
Current progress: not obvious improvement compared to strict filter according to evaluation loss. The conclusion may be different. Need more investigation on (be careful about the evaluation though.
[x] Reintegrate some elements from upstream.
[x] Add data packing for efficient training.
Bug: there are some problems about the packing. Since we use first_eos_indices to find the eos positions, we cannot integrate it with the packing because there are multiple eos tokens in the same input after data packing.
[ ] Need to either add the fine-tuning to Megatron or integrate some parallelism e.g. DeepSpeed.
[x] Integration with peft.
[x] Dry run with the first checkpoint from Santacoder large models.
Dataset
contenttoold_content,subjectandnew_content.Training
<NME> filename <....).strict filteraccording to evaluation loss. The conclusion may be different. Need more investigation on (be careful about the evaluation though.first_eos_indicesto find theeospositions, we cannot integrate it with the packing because there are multipleeostokens in the same input after data packing.peft.LAWAmethod to integrate different checkpoints for a better performance (https://github.com/JeanKaddour/LAWA).dedupversion ofpjjdataset.Idea
commit-datasetwill be useful for their downstream datasets.Debug
nanproblem when training with theverb-filtersetting (only when validation).