Improve memory management for cache loading

[GoogleCodeExporter]

Loading icecache files fills the heap pretty fast due to the nature of the 
icecache data sets: it can get potentially too huge to load everything 
in-memory. Here's some ideas for improving this:

- compress cache files in memory and uncompress current cache to display in 
OpenGL
- use pytables to implement a memory/disk caching

Original issue reported on code.google.com by mab...@gmail.com on 23 Jan 2011 at 9:18

[GoogleCodeExporter]

Multiple approach have been experimented over the last few months:
1-multi-process loading + shared memory
-files are dispatched to available cores and data is read by the main app 
through a shared memory buffer. The problem with this approach is that the 
shared memory buffer size has to be defined by the main app before loading. A 
big memory buffer would have to be declared upfront which was not adequate.
2-multi-process loading + shared memory + socket:
-similar to #1 but instead of pre-declaring the memory buffer, a socket was 
opened by each process after reading a file to tell the main app to allocate a 
proper buffer size. Once allocated, the process fills the memory buffer with 
the data and notify the main app when the data is ready to be read. Worked ok 
but it's a bit complicated to implement and to maintain. This solution is 
pretty fast though (~ 6x) compared to the unique cpu approach. Still the 
solution doesn't improved the memory management for huge giles, which put the 
application to an halt when the overall available memory is busted.
3-multi-process + HDF5:
-files are still loaded through multiple cores but converted to HDF5 on the 
fly. The main app is now all HDF5 centric, all memory management is done by 
HDF5 which allows pretty huge files to be loaded. The speed is not as good as 
#2 but stiil fine. 

So solution #3 has been implemented and checked-in. H5py is used as HDF5 
bindings. It is easy to use, fast and does a very good job at memory management.

Original comment by mab...@gmail.com on 9 May 2011 at 2:59

• Changed state: Fixed