More comprehensive multiprocessing

ratan-lab / sumo

Subtyping tool for multi-omic data

https://pypi.org/project/python-sumo

MIT License

13 stars 1 forks source link

More comprehensive multiprocessing #2

Open sienkie opened 5 years ago

sienkie commented 5 years ago

TODO:

[ ] Add multiprocessing to "prepare" mode, to have every network layer prepared in parallel. Make sure it performs well memory-wise for both high number of layers and large feature matrices.
[ ] Update multiprocessing in "run" mode. Currently we are running factorization in parallel for every sparsity value. If we run it on parallel for every (sparsity, repetition) pair, functions would be more atomic and we would make better use of resources.

sienkie commented 4 years ago

NOTE: "prepare" could also be made more memory efficient if we could delete every feature matrix from memory after the corresponding adjacency matrix was created