When loading the tokenizer: ModuleNotFoundError: No module named 'transformers_modules.baichuan_0'

[zouyingcao]

First, I load the tokenizer using the original file from hf model_path_baichuan = '../Baichuan2-7B-Base' tokenizer_baichuan = AutoTokenizer.from_pretrained(model_path_baichuan, trust_remote_code=True) When I just save the tokenizer using tokenizer_baichuan.save_pretrained(output_folder) and load it again from the output_folder tokenizer = AutoTokenizer.from_pretrained(output_folder, trust_remote_code=True)

Issue comes: ModuleNotFoundError Traceback (most recent call last)

----> tokenizer = AutoTokenizer.from_pretrained(output_folder, trust_remote_code=True)

ModuleNotFoundError: No module named 'transformers_modules.baichuan_0