Add cache for LDAP certificates

MrMikeFloyd commented 3 years ago

Currently, secon-tool supports loading certificates from LDAP servers directly. This works well for environments where highly available access to at least 1 LDAP server can be ensured. In order to improve operability in environments where LDAP availability is not guaranteed to be highly available it would be desirable to have cached certificate data available. This would allow functionality even in case of temporarily unavailable LDAP servers. As certificate data for a given organisation changes only every few years, there is relatively low risk for using outdated certificate information.

MrMikeFloyd commented 3 years ago

In case caching is something that would be considered a helpful feature we'd be happy to provide the functionality for this (e.g., create a pull request).

christian-schlichtherle commented 3 years ago

We have discussed this topic. Our thinking is that this should be implemented as a LRU heap cache (maybe using LinkedHashMap) using the decorator pattern for the Directory interface. The facade class SECON should then provide another method to decorate a given Directory object. So then, a user could simply mix a cache into the configuration of the subscriber by calling this method.

If you have such an implementation, your merge request would be very welcome. Otherwise, let's discuss your design.

MrMikeFloyd commented 3 years ago

Your proposed approach sounds like a good idea. Just a little background on what we used so far (and why): We started our development with an older version of the SECON library that didn't yet support retrieving public keys from LDAP. Also, inside our environment we had several LDAP directories for private keys, none of them being highly available. For that reason, we implemented our own LDAP client and decorated that with a cache for 2 reasons:

To circumvent the fact that we have several non highly available LDAP servers our client queries all of the available servers, until the first non-error response is received
To limit the number of LDAP calls and circumvent a situation where none of the servers might be available, we implemented a cache.

For caching, we decorated our LDAP client with a read-through implementation built upon caffeine. Here, we use a two-phase eviction strategy:

Asynchronously refresh public keys that haven't been fetched for a while. If that fails, keep the last known entry
Evict public keys after they haven't been fetched (i.e. used) for a while

This code snippet might make it more clear how the cache is configured (pretty standard, nothing exotic here):

Caffeine.newBuilder()
        .ticker(ticker) // optional time source
        .refreshAfterWrite(ldapCacheConfiguration.getRefreshAfter()) // triggers asynchronous refresh of stale entries
        .expireAfterWrite(ldapCacheConfiguration.getExpireAfter()) // evicts expired entries
        .build(ldapClient::findCertificates); // the decorated ldap call

Since this is something that we used outside of SECON, this would need to be adjusted to match the call structure of the SECON library. So far though, our experience with caffeine was quite good. Sorry for the wall of text, I just wanted to make our intent clear. I hope this is helpful! In case you'd like to further discuss possible implementation approaches, please let me know.

Thanks, and a happy new year to you!

DieTechniker / secon-tool

Add cache for LDAP certificates #36