willyborankin
commented
1 year ago Hi @nknize, @andrross, and @reta I'm the author of the encrypted-repository-opensearch plugin.
Below you will find some thoughts on what I believe OpenSearch is missing as a default solution for plugins that use encryption.
Beside I would like to invite opensearch-project/security team: @scrawfor99 , @peternied, @cwperks, @cliu123, @DarshitChanpura, @davidlago to discuss this solution as well.
All repository plugins, as you all know, just work with stream chunks, which the OpenSearch kernel sends to 3rd-party systems like HDFS, file systems, clouds, etc. To avoid different configurations of how they should encrypt data. I suggest using the KEK/DEK approach.
What is DEK/KEK
Data Encryption Key (DEK) The key used to encrypt data itself is called a data encryption key (DEK).
Key encryption keys (KEK) The DEK is encrypted (also known as wrapped) by a key encryption key (KEK). The process of encrypting a key with another key is known as envelope encryption.
KEK management rules
- Store KEKs centrally
- You could use a single KEK to wrap all DEKs that are responsible for that workload's encryption.
- Rotate keys regularly, and also after a suspected incident
DEK management rules
- Generate DEKs locally
- When stored, always ensure DEKs are encrypted at rest
- For easy access, store the DEK near the data that it encrypts
- Generate a new DEK every time you write the data. This means you don't need to rotate the DEKs
- Do not use the same DEK to encrypt data from two different users
- Use a strong algorithm such as 256-bit Advanced Encryption Standard (AES) in Galois Counter Mode (GCM).
For that, OpenSearch needs to have a Key Manager Service, which plugins will use to generate DEK keys.
Key Management Service
The key management service is a central service which OpenSearch provides to plugins to use for encryption. It can be integrated with 3rd party KMS e.g. Amazon, Google etc. There is only one responsibility for the key management service: to generate, encrypt, and decrypt DEK keys or additional data on demand. It can be configured to use different key management systems using names, so that plug-ins can use different key management systems. The configuration of the key management service needs to be done via the OpenSearch key store and shouldn't affect the bootstrapping of nodes or cluster formation. The default implementation uses RSA-256 bit keys to encrypt/decrypt AES-256 bit DEK keys and act as KEK.
The POC interface:
interface KeyManagerServiceProvider {
default KeyManagerService keyManagerService() {
return keyManagerService(“default”);
}
KeyManagerService keyManagerService(final String name);
}
interface KeyManagerService {
Supplier<SecretKey> dekGenerator();
byte[] decrypt(final byte[] data)
byte[] encrypt(final byte[] data);
}Keys rotation:
- The KEK/DEK model forces to use multiple DEK keys for encryption and decryption so rotation for DEK keys is working out of the box
- KEK keys rotation depends on the key management service configuration. In case of 3rd party KMS providers it is up to the provider. For the default key management service the rules are the same as for all other security settings: update settings on all nodes in the cluster and call
/_nodes/reload_secure_settingsor/_nodes/<node_id>/reload_secure_settingson the management node.
Default Key Management Service Configuration Key Store Settings
key.management.service.default.public.key- RSA 256-bit public keykey.management.service.default.private.key- RSA 256-bit private keykey.management.service.default.dek.encryption.algorithm- RSA encryption/decryption algorithm
Key Management Service Configuration Key Store Settings For 3rd Party Providers
key.management.service.<name>.<provider>.*- set up all necessary 3rd provider settings to use it
All management services can be configured independently.
How it will look like for plugins
In case of Key Management Service all plugins will use the same rules/recommendations for encryption: the main is "Generate a new DEK every time you write the data" which means DEK key per stream.
E.g. in case of encrypted-repository-opensearch all you need to add is only KMS name which will be use by the plugin to generate DEKs:
PUT _snapshot/repository_name
{
"type": "encrypted",
"settings": {
"storage_type": "fs",
"key_manager_name: "some_kms",
"location": "/mount/backups/my_fs_backup_location"
}
}and during configuration of the repository ClusterService will return information about the OpenSearch key manager service the plugin needs to use and after that generate DEK for each stream.
If I'm missing something fill free to point out.
P.S. In the end I would like to address some comments I read in PRs about the encrypted-repository-opensearch plugin which I found unfair and will address here:
It does not support partial encryptionTBH, I do not understand what partial encryption is. If it is about blocking encryption, I am afraid it was compromised; e.g., AES/CBC are not recommended for such encryption. Yes, it is possible to implement a custom solution for such case which again needs to be proven stable and will look like CBC. If it is about AES/GCS and the limitation of the stream size up to 64GB due to the IV and the fact that it must be renewed, well, it's a natural limitation of AES/GCM which is solved via the hard limit of a chuck size of 64 GB in the plugin since the default chuck size for all clouds is 5 TB, and in this case the whole stream will be compromised.
It does not support chucking It does support chunking this solution was select this way since it is responsibility of OpenSearch (in the past ES) to split streams into chunks which it does perfectly fine.
It is a custom wrapper around java encryption libraryI do not understand what a custom wrapper in this case. The plugin uses classical algorithms and cryptographic primitives e.g. AES 256-bit keys in Galois Counter Mode which were proved to be stable for such tasks. The only reason why BC was selected is that the problem of JDK which does not support streaming with more than 2GB and it is only one well known and trusted solution in opensource world. It is possible to use AWS or Google libraries, but I do not know how popular are they and it is up to the OpenSearch team.If it is about storing IV and encrypted data at the beginning of the stream, again, it is just an optimization. In this case, you don't need to do an additional network round trip.
it is tightly coupled with repository pluginYes and it is only right solution since OpenSearch core controls when it wants to send data somewhere. Moving such CPU bounded tasks to the core will affect performance of OpenSerach.
reta
andrross
peternied
vikasvb90
The repository interface in OpenSearch enables storing snapshot data in an external object store. It is implemented as a plugin interface, and the OpenSearch bundle by default ships with multiple plugin implementations (e.g. Google Cloud Storage, Amazon S3, Microsoft Azure). These object stores may offer server-side encryption, but OpenSearch itself offers no out-of-the-box mechanism to enable client-side encryption of the snapshot data. Both Amazon OpenSearch Service and Aiven have implemented client-side encryption, so there is clearly a need for this feature. The request here is to offer first class support for this feature within the OpenSearch Project.
We have multiple projects in flight (remote-backed index, searchable snapshots, etc) that are building on the repository interface. All of these stand to benefit from a client-side encryption feature.
Describe the solution you'd like The Aiven implementation is an open source plugin that essentially wraps the repository interface, allowing users to compose the encryption plugin with the object store implementation of their choice. An obvious option here is to bring this plugin along side the other object store plugins that are bundled in the OpenSearch distribution.
Describe alternatives you've considered There might be benefits to building client-side encryption more natively into the repository interface. The compose-ability of the Aiven implementation offers great flexibility, but it does mean that users must be aware of the plugin and configure it accordingly. Building it more directly into the interface could have usability benefits?