search

Blosc / bcolz

A columnar data container that can be compressed.
http://bcolz.blosc.org
959 stars 149 forks source link

how to share ctable object in share memory with multi process #412

Open WoolenWang opened 3 years ago

WoolenWang commented 3 years ago 
the_ctable = bcolz.open('test_ctable.bcolz', 'r')
table_ndarray = the_ctable[:]
from multiprocessing.sharedctypes import RawArray
the_memory = RawArray('b', table_ndarray.nbytes)
dst_data = np.ndarray(buffer=the_memory, dtype=table_ndarray.dtype, shape=table_ndarray.shape)
np.copyto(dst_data, table_ndarray)
from multiprocessing import Process
proc = Process(target=other_process,args=(the_memory,))
proc.start()

using this way can share the ctable data with ndarray type, but the data is not compressed and consume too many memory, is that any way to share ctable object in share memory ??

kaybinwong commented 3 years ago

you can take a shot on shared-array, good luck.

how to read data from carray, do u kown?

WoolenWang commented 3 years ago

you can take a shot on shared-array, good luck.

how to read data from carray, do u kown?

the most important thing is the numpy ndarray is not compressed and the bcolz ctable is compressed in memory so the usage of memory is significant different . the shared-array pkg is still try to share the numpy ndarray ?