This warning isn't helpful (and is confusing for the finetuning case). Only text dataloader accepts eos/bos token id, to be used for per sequence attention masking on pretokenized and concatenated sequences. Essentially all tokenizers have an eos/bos, and if you use mpt with attn_uses_sequence_id (to enable per sequence masking) without an eos/bos specified, you will get an error separately from this one. finetuning dataloader does not accept this.
This warning isn't helpful (and is confusing for the finetuning case). Only
textdataloader accepts eos/bos token id, to be used for per sequence attention masking on pretokenized and concatenated sequences. Essentially all tokenizers have an eos/bos, and if you use mpt withattn_uses_sequence_id(to enable per sequence masking) without an eos/bos specified, you will get an error separately from this one.finetuningdataloader does not accept this.