MemoryBlobStore limits the delegate store to a single thread

[brettniven]

The MemoryBlobStore limits the delegate store to a single thread, which can degrade performance. It effectively forces all 'put', 'get' and other operations to be bound to a single queue.

I've written a sample test and a one-liner change to highlight the problem. This is here: https://github.com/brettniven/geowebcache/commit/b7f08e2567fb10a7bdd81fca32953fe503c9d876. By increasing the executor threads, it obviously has much better performance.

I can assist on a fix - I'm just not sure on the best approach as I can't figure out why it would have the '1 thread' limit in the first place. Some links to explain what is happening:

• MemoryBlobStore constructs executorService with a pool limit of 1: https://github.com/GeoWebCache/geowebcache/blob/fb012452fafcc19ea062159c1e0e722586e16e16/geowebcache/core/src/main/java/org/geowebcache/storage/blobstore/memory/MemoryBlobStore.java#L99
• All put/get/other operations invoke executeBlobStoreTask, which waits, here: https://github.com/GeoWebCache/geowebcache/blob/fb012452fafcc19ea062159c1e0e722586e16e16/geowebcache/core/src/main/java/org/geowebcache/storage/blobstore/memory/MemoryBlobStore.java#L611
• The executorService is documented in comments here: https://github.com/GeoWebCache/geowebcache/blob/fb012452fafcc19ea062159c1e0e722586e16e16/geowebcache/core/src/main/java/org/geowebcache/storage/blobstore/memory/MemoryBlobStore.java#L52

... but I can't work out a reason for it needing to be limited to 1 thread. Each delegate store should have it's own locking code, where applicable.

For reference, our setup is:

• GWC embedded in GeoServer, 1.24.x, using Kartoza docker image
• We're hosting in Azure, using an Azure File Share (which unfortunately for us, is 'slow' for a few potential reasons, which exacerbates this issue - we're in the process of trialling the Azure Blob Store plugin)
• We're using a MemoryBlobStore as a 2-layer cache, fronting a FileBlobStore. i.e. innerCachingEnabled=true.
• We investigated performance and found that we get substantially better performance when we turn the MemoryBlobStore off, which is a very simple workaround for this issue. However... we believe we've also found a thread deadlock issue in the FileBlobStore code (which I'll raise in another issue when time permits)

[aaime]

@brettniven I don't know either (some research would be needed) but:

• I also have the impression the code is over-using locks and concurrency limitations
• The class is 10 years old
• The original author was well-meaning, but also inexperienced. Maybe he wanted to avoid potential double-computation of the same tile? But that's the job of other parts of GWC.

Generally speaking, AFAIK a BlobStore implementation should be thread-safe, and a wrapper should not need to care for synchronizing/serializing access to it. @smithkm can you provide any more information/insight?