search

mistralai / mistral-inference

Official inference library for Mistral models
https://mistral.ai/
Apache License 2.0
9.58k stars 846 forks source link

PAD token missing ? #150

Closed omkar-12bits closed 4 months ago

omkar-12bits commented 5 months ago

i tried using eos_token , unk_token and bos_token with left and right padding side but whenever the padding tokens size increases the outputs are pure garbage.

omkar-12bits commented 4 months ago

after trying this on both mistral and mixtral its sure that paddings doesn't work very well with these models. i was just playing with prompts at inference time and saw that paddings makes generation worst. if inference on batches doesn't work well then how does it performs while training ? shouldn't it also produce garbage ?

geronimi73 commented 4 months ago

can you post the code you use please?