search

rainwoodman / sharedmem

A different flavor of multiprocessing in Python
GNU General Public License v3.0
81 stars 9 forks source link

support for sparse matrices and python dict #1

Open sashamigdal opened 11 years ago

sashamigdal commented 11 years ago

Most of large objects I use in my code are either SparseMatrix from scipy.sparse or python dict. In both cases there are numpy arrays inside these objects and both objects could be constructed from arrays, such as: src_matrix((data,(row, col)), shape, copy=False) dict(zip(keys, values)) where data, row, col, keys, values can be used as numpy arrays. These arrays can be made sharedmem arrays, but this is not automatic. I have some large code already written before I learned about this package. Now I have to rewrite the whole code by wrapping every array into sharedmem.copy() inside Sparse Matrices and dicts. It would be nice to have these objects supprted in sharedmem along with ordinary numpy arrays. Then Slave/Master would produce and pass to Master/Slave the sparse matrices and dicts with shared memory, without any modifications of original code

rainwoodman commented 11 years ago

Would you mind share an example how the rewrite is done?

rainwoodman commented 8 years ago

@sashamigdal did you find a work around to the issue? Are you still manually converting the types of arrays? If there are no updates I will close this issue. It has been two years.

rainwoodman commented 5 years ago

Python 3.8 introduced sys-V style shared memory, and provides a 'shared' List class for primitive types: https://docs.python.org/3/library/multiprocessing.shared_memory.html