search

open-iscsi / tcmu-runner

A daemon that handles the userspace side of the LIO TCM-User backstore.
Apache License 2.0
189 stars 148 forks source link

RBD cache in write-through mode #479

Open emmericp opened 6 years ago

emmericp commented 6 years ago

The RBD cache is currently completely disabled as writeback caching obviously wouldn't work without implementing a proper replicated cache. However, using the cache in writethrough mode to cache reads should be safe.

Background: some poorly behaved hypervisors believe that 64kb IOs with a queue depth of 1 is an appropriate way to read large files sequentially. Patching out the hard cache disable and configuring it to act as a write-through cache with generous read-ahead helps a lot here.

Any reason why this wouldn't be safe? As in: is there a failure mode where another gateway can become active for writes and the primary doesn't lose its cache? This shouldn't happen because the other one can only be active if the primary loses the rbd exclusive lock implying the cache is lost (a problem for a writeback cache, but not a write-through cache).

mikechristie commented 6 years ago

I think as long as the cache is invalidated when we take the lock or open the image then write-through would work ok like you suggest.

I see some invalidate calls, but I have not had time to check the code thoroughly or test it out. Have you done this already?

mikechristie commented 6 years ago

Was also going to add, if this is not done automatically for us, then when we grab the lock we could just add our own invalidate calls.

emmericp commented 6 years ago

Yes, I did test it and looked at the code; but not thoroughly.

Hoped that someone with some more knowledge about librbd could chime in here. My understanding is that it's up to librbd to handle that, it cannot rely on its cache if the lock was lost and re-acquired. And yeah, there are some calls to invalidate the cache related to exclusive locking, but I'm not 100% sure about the control flow here.