Closed hermands closed 9 years ago
Don't know until we try but I think that we can retrieve all extracted features into a dense array with reasonable memory requirements. HDF5 is good but it is still orders of magnitude slower than RAM.
On Thu, Jan 1, 2015 at 11:50 AM, Daniel Herman notifications@github.com wrote:
Assigned #33 https://github.com/davidpng/FCS_Database/issues/33 to @davidpng https://github.com/davidpng.
— Reply to this email directly or view it on GitHub https://github.com/davidpng/FCS_Database/issues/33#event-213325718.
HDF5 and h5py appear to have good support for parallelization, but require specific installation options.
I imagine this would improve FCS data processing considerably, but perhaps not worth it if we are only doing it a few times.
What about retrieving data from HDF5 object? Are we loading all into memory -- how big are we talking? Or do we want parallel access?