Closed GoogleCodeExporter closed 8 years ago
Multiple approach have been experimented over the last few months:
1-multi-process loading + shared memory
-files are dispatched to available cores and data is read by the main app
through a shared memory buffer. The problem with this approach is that the
shared memory buffer size has to be defined by the main app before loading. A
big memory buffer would have to be declared upfront which was not adequate.
2-multi-process loading + shared memory + socket:
-similar to #1 but instead of pre-declaring the memory buffer, a socket was
opened by each process after reading a file to tell the main app to allocate a
proper buffer size. Once allocated, the process fills the memory buffer with
the data and notify the main app when the data is ready to be read. Worked ok
but it's a bit complicated to implement and to maintain. This solution is
pretty fast though (~ 6x) compared to the unique cpu approach. Still the
solution doesn't improved the memory management for huge giles, which put the
application to an halt when the overall available memory is busted.
3-multi-process + HDF5:
-files are still loaded through multiple cores but converted to HDF5 on the
fly. The main app is now all HDF5 centric, all memory management is done by
HDF5 which allows pretty huge files to be loaded. The speed is not as good as
#2 but stiil fine.
So solution #3 has been implemented and checked-in. H5py is used as HDF5
bindings. It is easy to use, fast and does a very good job at memory management.
Original comment by mab...@gmail.com
on 9 May 2011 at 2:59
Original issue reported on code.google.com by
mab...@gmail.com
on 23 Jan 2011 at 9:18