pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
https://pathway.com
Other
2.84k stars 98 forks source link

[Bug]: `pw.run()` never terminates when used with `multiprocessing` #54

Open KamilPiechowiak opened 1 month ago

KamilPiechowiak commented 1 month ago

Steps to reproduce

This simple program never terminates:

import pathway as pw
import multiprocessing

def local_run():
    pw.run(monitoring_level=pw.MonitoringLevel.NONE)

p = multiprocessing.Process(target=local_run)
p.start()
p.join()

(local_run added to disable monitoring) It is caused by a deadlock on logger's flush. This receiver never receives the message it waits for. Disabling the logger flushing makes program terminate correctly.

Relevant log output

no output

What did you expect to happen?

The program to finish.

Version

0.11.1

Docker Versions (if used)

No response

OS

Linux

On which CPU architecture did you run Pathway?

x86-64