HIT-SCIR / ltp

Language Technology Platform
http://ltp.ai
4.93k stars 1.04k forks source link

Abnormal behavior with the multiprocessing package #483

Open linfeng-du opened 3 years ago

linfeng-du commented 3 years ago

I have tried on multiple machines and it seems that when adding the importation code, running Pool workers will take up a huge amount of memory and very likely to cause an OOM :( Even for the simple code snippet as below (the importation code itself doesn't seem to be very memory consuming).

from multiprocessing import Pool
from ltp import LTP

def f(x):
    return x * x

if __name__ == '__main__':
    with Pool() as pool:
        pool.map(f, range(10))

I'm wondering if you could reproduce this problem 0.0.

AlongWY commented 3 years ago

I test your code and reproduce the problem and I will fix it soon. Hope everything goes well.

WeChat450c7a6e1f515cd2f9dbc4a3b637a02e