Open Atharva-Phatak opened 1 month ago
I went through your codebase, could you please redirect me as to how have you prepared data in aspects of tokenization ? Did you add a bos token ? Did you concat sequences together with an eos token ?
An example would be amazing.
I went through your codebase, could you please redirect me as to how have you prepared data in aspects of tokenization ? Did you add a bos token ? Did you concat sequences together with an eos token ?
An example would be amazing.