Instead of keeping all data in memory and writing it to a pickle file at the end of random network generation or stat computation, each process writes its results to a new file in a temporary directory. Then, at the end of a script, this directory is zipped.
Instead of keeping all data in memory and writing it to a pickle file at the end of random network generation or stat computation, each process writes its results to a new file in a temporary directory. Then, at the end of a script, this directory is zipped.