[Feature Request] Azure-repository snapshot support Client-side encryption for blobs

[cthtrifork]

Is your feature request related to a problem? Please describe

We would like the plugin to support client side encryption: https://learn.microsoft.com/en-us/azure/storage/blobs/client-side-encryption?tabs=java

Is there support for adopting this feature? We might make the pull request but I wanted to hear the opinion of the maintainers.

Describe the solution you'd like

Support the optional setup for client side encryption and allow configuration of access to Azure Key Vault.

Related component

Plugins

Describe alternatives you've considered

No response

Additional context

The Azure Blob Storage client libraries use envelope encryption to encrypt and decrypt your data on the client side. Envelope encryption encrypts a key with one or more additional keys.

The Blob Storage client libraries rely on Azure Key Vault to protect the keys that are used for client-side encryption. For more information about Azure Key Vault, see What is Azure Key Vault?.

[cthtrifork]

I think we should also look into support managed identities for Azure Key Vault as well. As it became supported in https://github.com/opensearch-project/OpenSearch/issues/12423

[andrross]

[Triage - attendees 1 2 3 4 5]

Thanks for filing. We do have work in flight for general client-side encryption for all repository stores: #5800, #7229

We could also explore the option of adding the option to use the feature within repository-azure as well as an alternative. I'm not totally opposed to that, but would be happy to hear the opinion from others as well.

/cc @vikasvb90

[vikasvb90]

@cthtrifork We rolled out client side encryption support outside of the repository plugins on top of repository layer so that encryption and repository plugins remain decoupled. The whole framework is divided in two parts:

1. Crypto Key provider plugin: A plugin of this type is responsible for providing data key pairs to be used in encryption/decryption. There is a AWS KMS based crypto-kms plugin available which retrieves keys from AWS KMS service. Similar plugins can be added for other master key providers.
2. Encryption/Decryption layer: A crypto abstraction plugin is available in OpenSearch here to encrypt or decrypt provided streams using the configured key provider against a repository. Unfortunately, there is no default plugin available for this yet and hence, you may need to build and keep a custom implementation. Once this plugin or module is available, this gets loaded by registry against the configured encrypted repository.

Once, both of these components are available, you can create an encrypted repository. Following is how a sample request would look like.

PUT _snapshot/vikasvb-repository-test-1
{
  "type": "s3",
  "settings": { // Repository settings
    "bucket": "<bucket-name>",
    "region": "<region>"
  },    
  "crypto_settings": {
      "key_provider_name": "user-db-repository-key",
      "key_provider_type": "aws-kms", // Key Provider Type
      "settings": { // Key provider settings 
          "key-1": "sample value"
      }
  }
}

key_provider_type here is the identifier you will configure for your key provider impl. aws-kms is the type for the crypto-kms plugin as mentioned here. key_provider_name is an identifier used in registration of a master key for a given key provider type. You can use the same combination of key provider name and key provider type against any repository or you can configure different keys depending on your use case.

[cthtrifork]

@vikasvb90 That is great. I don't think it make sense to do a proprietary/vendor solution if there is upcoming support for a generic approach. Any thoughts on when this could be available? Summer? Fall? Winter?