Unsafe multiprocessing

[AngusMingare]

Description

Several places in Symmer use multiprocessing in the following way:

with mp.Pool(mp.cpu_count()) as pool: 

New process are created in one of two ways:

• Forking: default on linux and older MacOS systems
• Spawning: default on windows and newer MacOS Spawning a new process causes the top-level module to be re-imported to the new process which can cause infinite loops spawning new processes.

Minimal example

Running Symmer on MacOS Monterey (Apple M1 chip), I encountered this problem trying to run the code in notebook 2.2 as a local python script after cloning the Symmer repo to my local machine.

Suggested fixes

Ensuring all experiments run using Symmer are contained in the

if __name__ == "__main__":

idiom will prevent the top-level script being executed when it is imported to the spawned process stopping the infinite loop issue. This should then be documented in Symmer.

Alternatively, it's possible to force the new process to fork by using:

with mp.get_context("fork").Pool(mp.cpu_count()) as pool: 

This also fixes the issue without requiring the above if name == main idiom. However this may be bad practice (?). It would also need to be tested on Windows OS.

[AlexisRalli]

This is a known issue. We are considering moving to the ray library , this should fix the problem and may give other benefits too (due to shared objs not being pickled).

[AlexisRalli]

Fixed in final pull requests: #151 and #153