Open Harry1035 opened 6 months ago
"ValueError: cannot find context for 'fork' " I got the same problem
Thanks for your solution about switching to "spawn" mode! I am having the same issue on windows. I tried several ways but none of them work. I guess the problem lies in reduce = Process(target=reduce_process, args=(output_queue, output))
, where the file output
is a non-pickable parameter.
See here.
You could try running this code in ubuntu or other linux system.
In the windows environment, the following problems occur when extracting the WIKI corpus:
dit: wikiextractor\WikiExtractor.py line 417
Process = get_context("fork").Process -> Process = get_context("spawn").Process
A new problem arises: