[x] Refactoring the output format of the dataset to support flexible training & evaluation.
Main change: from content to old_content, subject and new_content.
Motivation: we should know the input length when calculating the loss over the target, this is necessary for GPT-like model fine-tuning.
[x] Add hyper-parameter information to the dataset to make it clear for each dataset.
Main change: write into a file to record the hyper-parameter for different datasets.
Motivation: this is to double-check the hyper-parameters of different filtering strategies, and then figure out the best one with the experimental results.
Training
[x] Adding the argument which enables the user to remove the loss in the input buggy code & commit message.
Conclusion: this kind of modification will reduce the loss value on train and dev by a large margin, which indicates that the previous wired train loss curve may be caused by the improper loss computation over inputs also. (Still verifying).
[x] Add the support for merging examples from the subset of FLAN.
[ ] Add support for evaluation on different benchmarks during pre-training after every X steps.
[x] Add the option to enable computing loss on instruction + target but not in the buggy code.
[x] Given the old_content and commit_message , just output the diff instead of the while file. It may make the evaluation harder since we should apply these diffs into the original content to obtain the new_content. But it has an advantage on that the old_content can be larger.
Progress: the current diff is obtained via a string-level comparison. May be worthy to try using line-level comparison.
[ ] Their experiments show that truncation on the file content (e.g., cutting the old_content to some relevant lines to make it shorter) will result in worse performance. Our current implementation mixes these two modes: in 20% time, the model will accept the whole file content as the input; in 80% time, the model will randomly cut the input file content into a reasonable range. Maybe 100% whole file content will be better?
[x] Experiment if adding filename is better (In Carper they do <NME> filename <....).
Current progress: not obvious improvement compared to strict filter according to evaluation loss. The conclusion may be different. Need more investigation on (be careful about the evaluation though.
[x] Reintegrate some elements from upstream.
[x] Add data packing for efficient training.
Bug: there are some problems about the packing. Since we use first_eos_indices to find the eos positions, we cannot integrate it with the packing because there are multiple eos tokens in the same input after data packing.
[ ] Need to either add the fine-tuning to Megatron or integrate some parallelism e.g. DeepSpeed.
[x] Integration with peft.
[x] Dry run with the first checkpoint from Santacoder large models.
Dataset
content
toold_content
,subject
andnew_content
.Training
<NME> filename <....
).strict filter
according to evaluation loss. The conclusion may be different. Need more investigation on (be careful about the evaluation though.first_eos_indices
to find theeos
positions, we cannot integrate it with the packing because there are multipleeos
tokens in the same input after data packing.peft
.LAWA
method to integrate different checkpoints for a better performance (https://github.com/JeanKaddour/LAWA).dedup
version ofpjj
dataset.Idea
commit-dataset
will be useful for their downstream datasets.Debug
nan
problem when training with theverb-filter
setting (only when validation).