Closed MIracleyin closed 12 months ago
You must be running this on Windows, right? I'm sorry, but Windows does not support multiprocessing. You can only set workers=1, but this will be slower.
I'm afraid it's not a system issue, I am running this code in macOS.
Is the closure function worker_func can't put into processing Pool?
There are complete error log:
Count Ngrams: 0it [00:00, ?it/s]Traceback (most recent call last):
File "train.py", line 21, in
I've looked into it, and it seems that MacOS has the same problem as Windows, while Linux doesn't.
You can try:
workers=1
;import multiprocessing
then multiprocessing.set_start_method('fork')
;multiprocess
and then replace multiprocessing
with multiprocess
.Reference: https://medium.com/devopss-hole/python-multiprocessing-pickle-issue-e2d35ccf96a9
Thank you very match! The second way works fine for my Mac.
It was macOS using spawn
rather than fork
in the multiprocessing job.
This project is perfect for someone who wants to know word piece principles!
Hey, I had an error in my training.
but I got an error
AttributeError: Can't pickle local object 'Trainer.pcount.<locals>.worker_func'
This is my env version: python 3.8.16 multiprocess 0.70.14