[opinion] dynamic caching is more viable than counting memory consumption

[fabian-paul]

The implementation of the 'operate in memory' function in the coordinate module is complicated by the fact that it is impossible to reliably estimate the memory consumption in the transfromers. Each transformer calls one or many numpy or mdtraj functions each of which may allocate dynamic memory. Those allocations are rarely documentated and we are forced to inspect the source code of the libraries. These allocations can be of the same size as the chunk size.

Additionally the application runs in a multitasking operating system. So querying the amount of free memory and subtracting the amout of memory that our application is expected to consume is no reliable estimate for the free computer memory. Opening a tab in the Firefox browser may suddenly decrease the free memory by dozens of megabytes. Combined with our memory-greedy strategy of putting as many data into the RAM as possible this is a recipe for disaster.

One solution to this problem is that our application should have the abilty to 'back off' an return memory to the operating system in times of memory shortage. All operating systems have an API to report memory shortage to programs. We could interface this API and replace the 'operate in memory' by a data cache. When there is no memory shortage, all data is held in memory. In case of shortage, the cache is emptied and some chunks have to be recomputed.

Here are some possible guidelines for the implemention of such a cache:

• Transformers should not attempt to allocate one huge block of memory that will hold all data. Instead they should allocate a cache page for every chunk they recieve from their data producer. If there is no memory shortage, the transfomer can recover all the data from the cache. Example:

  # caching a chunk
  cache.insert(traj_number,frame_number,chunk)

• Cache pages that are not used at the moment can 'go away'. If the transfomer tries to fetch a chunk from the cache that has gone away, a KeyError is raised. The transfomer can then react to this by requesting the chunk again from its data producer. Example:

  # fetching a chunk
  try:
    chunk = cache.fetch(traj_number,frame_number)
  except KeyError: 
    # cache miss, get chunk from data producer 

• Deletion of cache pages happens as a rection to memory shortage. Memory shortage is reported directly from the operating system, see https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt http://www.newosxbook.com/articles/MemoryPressure.html In response, the cache can delete chunks that are currently not used. Whether a page is currently in use can be found out by checking the refcount of the numpy array which holds the cached chunk. The operating system notifications are received in a second (Python) thread. The cache is protected with a lock to prevent race conditions.
• On memory shortage, the cache attepts to delete those pages first, that have not been accessed for the longest time. The oldest chunks are typically the following:
  • chunks in chains that are not used but where the user still holds a reference
  • chunks from the first steps of a pipeline. When caching is enabled, these chunks are created first and are only accessed once or twice when the next transformer in the chain is parametrized and run. Therfore these are the oldest cached chunks.
  • The first frames from the first trajectories. They are always accessed before the chunks from the end of the trajectory. These chunks are easy to recover, because each transformer can reset its data producer and query the first chunks from the first trajectories again.

So this simplest form of caching which is based on access times, should give good performance when combined with the transformation pipeline.

[franknoe]

I think caching is a very good idea, but I would guess it is a major effort to do it. Am I right?

---

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[fabian-paul]

Trying this it out is not so difficult. Maybe we will run into some unforeseen problem like the Python garbage collector not giving memory back to the operating system etc. But this is hard to say in advance. The question is what is harder to do: implementing caching reliably or getting the old 'operate in memory' idea to work reliably?

[franknoe]

I really don't know. It's worthwhile to play around with it to get a feeling.

Am 26/03/15 um 19:51 schrieb fabian-paul:

Trying this it out is not so difficult. Maybe we will run into some unforeseen problem like the Python garbage collector not giving memory back to the operating system etc. But this is hard to say in advance. The question is what is harder to do: implementing caching reliably or getting the old 'operate in memory' idea to work reliably?

— Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/147#issuecomment-86667905.

---

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[marscher]

This might be interesting on building cache memory overflow callbacks on Linux.

Maybe there is a similar technique available on OSX.

https://pypi.python.org/pypi/cgroup-utils/0.5

cgroups provides a callback interface for userspace software to be informed if free memory will fall below a certain threshold.

So this is a nice possiblity to avoid swapping the system to death by maximizing chunksizes and amount of precomputed frames.

I strongly vote for having this possibility of only using a maximum memory threshold for cache clearance. Maybe we might also estimate cache hit rates analytically beforehand? This result can we use to rate the effort we wanna invest here.

But it really seems promising to avoid double computations with a fairly large cache!

[franknoe]

We should pick this up again after the API redesign. I am not sure if this is still an issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.