(Pre training mamba with train.sh) Error : GPT2BPETokenizer : assert args.vocab_file is not None

[SkanderBS2024]

Hello,

after pre-processing the dataset with a BPE tokenizer, when i launch the 'train.sh' script for mamba i do get this error.

In the script it's mentionned that i have to specify the tokenizer path, i did put the path for the "tokenizer.json" and when inspecting the code it expects the vocab.json and merge.json files does anyone have an idea about the path i should give as an argument ?

this is the command that launches the training script :

/workspace/megatron/examples/mamba# ./train.sh /workspace/megatron/examples/mamba/dataset/CleanData2/preprocessed2_text_document /workspace/megatron/examples/mamba/dataset/mamba_tokenizer/tokenizer.json

Assuming that CleanData2 contains the .bin and .idx processed data Assuming that mamba_tokenizer contains :

[tbsxxxH]

Hello,

after pre-processing the dataset with a BPE tokenizer, when i launch the 'train.sh' script for mamba i do get this error.

In the script it's mentionned that i have to specify the tokenizer path, i did put the path for the "tokenizer.json" and when inspecting the code it expects the vocab.json and merge.json files does anyone have an idea about the path i should give as an argument ?

this is the command that launches the training script :

/workspace/megatron/examples/mamba# ./train.sh /workspace/megatron/examples/mamba/dataset/CleanData2/preprocessed2_text_document /workspace/megatron/examples/mamba/dataset/mamba_tokenizer/tokenizer.json

Assuming that CleanData2 contains the .bin and .idx processed data Assuming that mamba_tokenizer contains :

Have you solved it? I also have this problem.

[SkanderBS2024]

@tbsxxxH

I've temporarily hard coded the paths here :

https://github.com/NVIDIA/Megatron-LM/blob/c873429cbaa43257d4d4fc01df2a7a50453b7984/megatron/training/tokenizer/tokenizer.py#L38-L40

[tbsxxxH]

@tbsxxxH

I've temporarily hard coded the paths here :

https://github.com/NVIDIA/Megatron-LM/blob/c873429cbaa43257d4d4fc01df2a7a50453b7984/megatron/training/tokenizer/tokenizer.py#L38-L40

How to do it specifically? I changed it to the following but still got an error:

        assert args.vocab_file is not None
        assert args.merge_file is not None
        tokenizer = _GPT2BPETokenizer('/home/eva.liu/Megatron-LM/vocab.json', '/home/eva.liu/Megatron-LM/merges.txt')

[SkanderBS2024]

remove the assertions and declare a variable for each path and give them as parameter to "_GPT2BPETokenizer"

[tbsxxxH]

Thank you, the problem is solved.

[zixianwang2022]

Hi, I think Mamba is using GPTSentencePieceTokenizer as described in the train.sh.

       --tokenizer-type GPTSentencePieceTokenizer \
       --tokenizer-model ${TOKENIZER_PATH} \

[github-actions[bot]]

Marking as stale. No activity in 60 days.