search

davidpng / FCS_Database

Program to scrape an FCS directory of metadata
GNU General Public License v3.0
3 stars 2 forks source link

Do we want HDF5/h5py to be parrallelable? #33

Closed hermands closed 9 years ago

hermands commented 9 years ago

HDF5 and h5py appear to have good support for parallelization, but require specific installation options.

I imagine this would improve FCS data processing considerably, but perhaps not worth it if we are only doing it a few times.

What about retrieving data from HDF5 object? Are we loading all into memory -- how big are we talking? Or do we want parallel access?

davidpng commented 9 years ago

Don't know until we try but I think that we can retrieve all extracted features into a dense array with reasonable memory requirements. HDF5 is good but it is still orders of magnitude slower than RAM.

On Thu, Jan 1, 2015 at 11:50 AM, Daniel Herman notifications@github.com wrote:

Assigned #33 https://github.com/davidpng/FCS_Database/issues/33 to @davidpng https://github.com/davidpng.

— Reply to this email directly or view it on GitHub https://github.com/davidpng/FCS_Database/issues/33#event-213325718.