fix several bugs (especially, the bug of preparing labels: the original code didn't deal with bos correctly, which made wrong labels; and the bug of the weird behavior of the tokenizer, see code comment for details)
supporting the dataset that is a mixture of visual-language data and pure-language data, which is needed for training on llava_v1_5_mix665k.json
add a data_collator to pad token_ids and labels for a batch, in this case, you don't need pad every batch to the length of max_seq_length, which would save gpu memory and speed up training
about model loading
remove the need to manually overwrite 'remote code'
add a toggle for disabling flash-attention 2
make lora module selection more elegant
divide the part of definition and the part of execution of the code train.py
The works I have done:
bos
correctly, which made wrong labels; and the bug of the weird behavior of the tokenizer, see code comment for details)llava_v1_5_mix665k.json
data_collator
topad token_ids
andlabels
for a batch, in this case, you don't need pad every batch to the length ofmax_seq_length
, which would save gpu memory and speed up trainingtrain.py