Open goodboy opened 4 years ago
Discussion surrounding this is now happening in python-trio/trio#1614 in terms of avoiding using a server all together which would actually be the most ideal.
Heh, and full conclusion is "write your own sync fork server" :laughing:
Motivating blog post: https://pythonspeed.com/articles/python-multiprocessing/
mulitprocessing
's forkserver has proven severely unreliable for nested process launching (#84, #134) however it is much much faster then our currenttrio
(aka async wrappedsubprocess
) spawning backend and, as such, we're keeping around for benchmarking purposes for when we eventually tackle the building our own internal subsystem for fast process launching on linux.The issues with current unix forking apis is covered in quite good depth in @catern's blog post and I think it's a multiple required read before diving into this.
I've had some (very) brief discussion and there's some ideas I'd like to investigate:
supervise_api
sfork()
mp
's fork request loop usingsfork
which does minimaltractor
templated process environment copying via a specific supervisor processI'd love to get @catern's feedback on a sensible direction with the fast forking sub-system design as well :surfer: