Key derivation API using the ledger secrets

[achamayou]

There are a number of contexts, such as producing commitments to private claims in receipts that require keys to be derived from a master secret. That master secret does not particularly need to change, but being able to re-key is always good.

We know from the experience of building the ledger secrets cache that supporting this in application is not trivial, and so rather than making each application re-implement it, it seems best to expose an API and allow keys to be derived from the ledger secrets. Re-keying then comes "for free".

An API such as:

// Write context
auto key = ctx->derive_key("label")
...
// Historical read context
auto key = ctx->derive_key("label", txid)

Would be ideal. In a write context, derive_key would need to create a read dependency on public:ccf.internal.historical_encrypted_ledger_secret, to accurately pick up ledger re-keys.

In a historical read context, the look up would use the txid to find the relevant ledger secrets and derive the key. Derivation would look like HKDF(app_prefix + user_label + ledger_secret).

[fournet]

This API would be very useful!

It provides a pseudo-random function (implicitly keyed by CCF), which can be used for deriving all kinds of secrets and reproducible randomness, not just keys. As an example, a CCF application may also use it as a replayable RNG.

I miss a parameter that indicates the requested number of bytes (N) in the derived secret.

I'd implement the derivation as HKDF(ledger_secret, SHA256(app_prefix) + SHA256(app_context) + N) or a similar formatting that prevents any collision on concatenations of variable_length arguments.

The ledger secrets would be stored in the KV and encrypted into the ledger, with the same confidentiality guarantees as private tables, and the same benefit when refreshing the ledger secret (enclaves compromised before the refresh don't get access to secrets derived afterwards).

[achamayou]

I miss a parameter that indicates the requested number of bytes (N) in the derived secret.

Indeed!

I'd implement the derivation as HKDF(ledger_secret, SHA256(app_prefix) + SHA256(app_context) + N) or a similar formatting that prevents any collision on concatenations of variable_length arguments.

Thank you for the suggestion!

The ledger secrets would be stored in the KV and encrypted into the ledger, with the same confidentiality guarantees as private tables, and the same benefit when refreshing the ledger secret (enclaves compromised before the refresh don't get access to secrets derived afterwards).

They are stored under public:ccf.internal.historical_encrypted_ledger_secret https://microsoft.github.io/CCF/main/audit/builtin_maps.html#historical-encrypted-ledger-secret, which is counter-intuitively public, but the contents of which are encrypted with the ledger secrets nonetheless.

This is to facilitate recovery, in which a node parses the public state of the ledger from the last snapshot, and builds a history of encrypted ledger secrets from it. Once recovery is complete, the reassembly of the wrapper key allows the node to decrypt the latest ledger secret, and work its way backward through the list of encrypted secrets from the last snapshot.

If the table was private, we wouldn't be able to extract that sequence during public state recovery, and we would need to decrypt transactions starting from the end, which is very impractical when rebuilding state.