Open liujiabin20810 opened 1 year ago
you said 'we can convert dataset in SFT_data/conversations to ./sft_data/train.jsonl Here is the code:' in the beginning, and then you said 'This method will not generate the train.jsonl, and it save the train&val dataset to train_data and val_data. It also can be loaded by class SFTDataset.' wtf?
you said 'we can convert dataset in SFT_data/conversations to ./sft_data/train.jsonl Here is the code:' in the beginning, and then you said 'This method will not generate the train.jsonl, and it save the train&val dataset to train_data and val_data. It also can be loaded by class SFTDataset.' wtf?
Maybe I didn't explain clearly. the load_data
method in finetune_moss.py
has two ways to load data, one way is reading dataset from jsonl type file and extract the content, saving to train_data and val_data. Another way is reading directly from train_data and val_data saved before.
The code above is saving the train_data and val_data , not train.jsonl.
you said 'we can convert dataset in SFT_data/conversations to ./sft_data/train.jsonl Here is the code:' in the beginning, and then you said 'This method will not generate the train.jsonl, and it save the train&val dataset to train_data and val_data. It also can be loaded by class SFTDataset.' wtf?
Maybe I didn't explain clearly. the
load_data
method infinetune_moss.py
has two ways to load data, one way is reading dataset from jsonl type file and extract the content, saving to train_data and val_data. Another way is reading directly from train_data and val_data saved before.The code above is saving the train_data and val_data , not train.jsonl.
I got you. I have successfully generated the train_data and val_data and finetuned my custom model. I would say that your shared code is extremely helpful! Thanks a lot!
we can convert dataset in
SFT_data/conversations
to./sft_data/train.jsonl
Here is the code:
This method will not generate the train.jsonl, and it save the train&val dataset to
train_data
andval_data
. It also can be loaded by classSFTDataset
.