google / ml-compiler-opt

Infrastructure for Machine Learning Guided Optimization (MLGO) in LLVM.
Apache License 2.0
629 stars 93 forks source link

[local-worker-mgr] Force using 'spawn' for process creation #307

Closed mtrofin closed 1 year ago

mtrofin commented 1 year ago

It turns out tflite conversion uses under the hood numexpr. Numexpr spawns a bunch of threads at module init. So if the main process converts a model, then forks and the resulting processes try converting a model, too, the state the forked processes work on appears like having been initialized. They hang waiting for results.

Using spawn and not forkserver because the latter still runs this risk. Spawning may be slower, but in the grand schema of things (training time), slightly larger init time is not that problematic.