Open dumpmemory opened 1 year ago
the tokenizer cant be hashed when using datasets.map function with num_proc >1 .
https://github.com/THUDM/ChatGLM-6B/issues/286
same problem, can anyone help to solve this?
https://stackoverflow.com/questions/55344376/how-to-import-protobuf-module Seems like protobuf is not picklable. I will look into it in next few days.
the tokenizer cant be hashed when using datasets.map function with num_proc >1 .
https://github.com/THUDM/ChatGLM-6B/issues/286