search

dabeaz / curio

Good Curio!
Other
4.02k stars 241 forks source link

Better subprocess wait() functionality #202

Closed dabeaz closed 7 years ago

dabeaz commented 7 years ago

The wait() method on Popen instances needs to be revamped. Right now it's a dumb sleep/retry poll on the corresponding function in the standard subprocess module. Not the most efficient. However, it's all a bit complicated in how this functionality gets mixed up with threads, the standard subprocess module, signals, async, and other things. Open to ideas.

njsmith commented 7 years ago

FWIW my plan in trio is to use system-specific APIs on Windows and kqueue systems, and on epoll systems just spawn a dedicated thread for each process to sit in waitpid. (I figure the overhead of spawning a thread is small compared to the cost of spawning a child, so this should be fine, and anything is better than SIGCHLD.)

dabeaz commented 7 years ago

I've thought about the idea of launching a background thread in Curio. Sort of wondering what the upper limit on active processes would be? Surely it would normally be far less than the number of threads that could be created.

A little curious about using system-specific APIs on Windows/kqueue though. What's the main upside of doing that as opposed to relying on subprocess?

dabeaz commented 7 years ago

I've changed the code to use a background thread for the moment. Might revise it later, but for now it will avoid the retry/poll loop.