celery / billiard

Multiprocessing Pool Extensions
Other
419 stars 252 forks source link

How to solve the race condition problem in Pool multiprocessing #357

Open jeffrey82221 opened 2 years ago

jeffrey82221 commented 2 years ago

Hi,

When I am getting and closing DB connection (psycopg2) in the processes of billiard Pool, I found that getting and closing of DB connection will fail. Is there any way that I can add some lock to the getting and closing methods such that the race condition problem can be avoid?

In multiprocessing, a Lock object is provided. I wonder if there are something similar in billiard.

from multiprocessing import Process, Lock

def f(l, i):
    l.acquire()
    try:
        print('hello world', i)
    finally:
        l.release()

if __name__ == '__main__':
    lock = Lock()

    for num in range(10):
        Process(target=f, args=(lock, num)).start()