Design memory mapping for X matrix

neurospin / pylearn-epac

Embarrassingly Parallel Array Computing: EPAC is a machine learning workflow builder.

BSD 3-Clause "New" or "Revised" License

12 stars 3 forks source link

Design memory mapping for X matrix #13

Open JinpengLI opened 11 years ago

JinpengLI commented 11 years ago

http://pythonhosted.org/joblib/

It seems this issue can be solved by the project, joblib.

JinpengLI commented 11 years ago

Thanks to the project joblib (http://pythonhosted.org/joblib/), the large matrix can be used as np.memmap in multi-processes on LocalEngine. c.f. the example run_a_big_matrix.py in the branch mmapping. After I come back, I will work for the memory mapping for the soma-workflow. @duchesnay @duboism