Closed oiabtt closed 7 months ago
Indeed adding the BOS / EOS / PAD tokens is intended to be done by the user. As in NLP, the common good practice is to use a data collator that will add them when creating the batches to be fed to a model. If you are using PyTorch, you can directly use miditok.pytorch_data.DataCollator.
I'm looking at the documentation and code, and I noticed that these special tokens ["PAD", "BOS", "EOS", "MASK"] are used in the configuration by default(https://miditok.readthedocs.io/en/v3.0.0/bases.html#special-tokens). However, during my experiments, I found that the generated tokens do not include BOS.