Training error by multiple works

[MIracleyin]

Hey, I had an error in my training.

trainer = Trainer(order=6, max_vocab_size=100000, min_count=32)
trainer.train(w, workers=2, batch_size=1000)

but I got an error AttributeError: Can't pickle local object 'Trainer.pcount.<locals>.worker_func'

This is my env version: python 3.8.16 multiprocess 0.70.14

[bojone]

You must be running this on Windows, right? I'm sorry, but Windows does not support multiprocessing. You can only set workers=1, but this will be slower.

[MIracleyin]

I'm afraid it's not a system issue, I am running this code in macOS. Is the closure function worker_func can't put into processing Pool? There are complete error log: Count Ngrams: 0it [00:00, ?it/s]Traceback (most recent call last): File "train.py", line 21, in trainer.train(w, workers=2, batch_size=1000) File "/Users/yin/forkrepo/bytepiece/bytepiece/bytepiece.py", line 163, in train self.ngrams = self.pcount_ngrams(texts1, workers, batch_size) File "/Users/yin/forkrepo/bytepiece/bytepiece/bytepiece.py", line 224, in pcount_ngrams return self.pcount( File "/Users/yin/forkrepo/bytepiece/bytepiece/bytepiece.py", line 199, in pcount pool = Pool(workers, worker_func, (in_queue, out_queue)) File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/context.py", line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/pool.py", line 212, in init self._repopulate_pool() File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool return self._repopulate_pool_static(self._ctx, self.Process, File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static w.start() File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/opt/homebrew/Caskroom/miniconda/base/envs/es/lib/python3.8/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'Trainer.pcount..worker_func' Count Ngrams: 0it [00:00, ?it/s]

[bojone]

I've looked into it, and it seems that MacOS has the same problem as Windows, while Linux doesn't.

You can try:

• workers=1;
• import multiprocessing then multiprocessing.set_start_method('fork');
• Install multiprocess and then replace multiprocessing with multiprocess.

Reference: https://medium.com/devopss-hole/python-multiprocessing-pickle-issue-e2d35ccf96a9

[MIracleyin]

Thank you very match! The second way works fine for my Mac. It was macOS using spawn rather than fork in the multiprocessing job. This project is perfect for someone who wants to know word piece principles!