Cache unwrapping keys to reduce load on key vault

[johanstokking]

Summary:

Cache unwrapping keys to reduce load on key vault

What do you see now?

Unwrapping keys on-the-fly, also inside loops.

What do you want to see instead?

Keys should not be stored (long) in memory, but they can be cached briefly within the same function.

---

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/1218 by @johanstokking

[johanstokking]

Please coordinate

[htdvisser]

Please coordinate. I don't have time for this.

[adriansmares]

I'll pick this up.

[adriansmares]

I can see two possible implementations with this one:

• Create a wrapper that takes a crypto.KeyVault and returns a crypto.KeyVault which caches the keys for a period of time upon unwrapping
  • This solution is transparent and requires no changes on the call sites
  • This will be reusable between gRPC calls since the keys will stay in memory
  • Since eviction is done based on time, keys may stay in memory longer than they are needed (but not too long)
• Create a wrapper around crypto.KeyVault only before 'mass usage', and defer the destruction/deletion of the keys from memory
  • This requires some changes to the call sites, but nothing extreme
  • Not reusable between separate gRPC calls since it would be out of scope
  • Keys stay in memory for the minimum amount of time

I'm more in favor of 1, but any thoughts on this @johanstokking ?

[johanstokking]

A few things on the context;

• Key wrapping serves two purposes: encryption at rest (by registries) and transferring keys encrypted between components
• The unwrapped keys are typically unique to sessions, so caching those is only relevant in a unit of work where they are used in multiple times, i.e. processing an uplink message that generates a downlink message, downlink queue operations, etc
• You can't easily define units of work in single routines; it's oftentimes a chain of events. For example, an uplink message that goes upstream (unwrap AppSKey for decryption) and upstream sends a downlink message back (unwrap AppSKey for encryption). Another example is NS sending a network-layer downlink, which invalidates the application queue so that AS needs to unwrap AppSKey for processing the uplink message as well as subsequent downlink queue invalidation events
• The cost is twofold: loading the KEK (depending on the key vault implementation) and the actual unwrapping. These are two separate things to optimize
• For caching:
  • The KEK value of a label won't change
  • The unwrapped session key of a session key ID won't change
  • So, you don't need active eviction on device activations or anything. The only reason to evict is reducing memory consumption

That being said, I prefer option 1. Those units of work last typically in the order of seconds. With https://github.com/TheThingsNetwork/lorawan-stack/issues/2708, we might get a bulk of messages of a single device, so having a cache expiry of 5 to 10 seconds seems reasonable, but please add some Prometheus metrics on the cache (hit ratio, size). Having a fixed size, long (hour) TTL and an ARC cache (balancing LFU and LRU) can also be just fine.