SalusaSecondus
commented
2 years ago Will this work with stock Java (not using BouncyCastle)? (I'm guessing that it will not.)
ACCP, like the crypto providers distributed by the OpenJDK, caches all AES-GCM plaintext and does not return any of it prior to completing the decryption. (In other words, it does not stream decryption.) This design decision was made to avoid "Release of Unverified Plaintext", a common security issue in many cryptographic systems.
Looking at the related issue, I see that you are talking about snapshots that are over 300GB. AES-GCM does not scale to encrypt data larger than about 64GB in a single call. Thus, you will need to fragment your files for successful decryption and cannot use AES-GCM directly. I strongly recommend you investigate the AWS Encryption SDK for Java which can encrypt/decrypt extremely large files in a streaming manner. This will:
- Let you protect files larger than ~64GB
- Let you decrypt in a streaming manner without needing to worry about the Release of Unverified Plaintext
- Let you decrypt in a streaming manner and thus avoid the memory overhead of caching the entire result
- Integrate with ACCP (for more efficient cryptography)
- Do all of the above while abstracting away many other cryptographic complexities and sharp edges.
willyborankin
Hi all, I'm trying to use Amazon Corretto together with OpenSearch to restore an encrypted snapshot and the way how Amazon Corretto allocates memory is IMHO is too aggressive. I have such use case:
OutOfMemorryError. AFAIU this happens due to the fact thatAccessibleByteArrayOutputStreamincreases its memory using amortizer as result to restore 4GB index (OpenSearch uses a thread pool for that in this particular case 3 threads) I need to set up node heap size at least 10GB, which is not so good. So the question is: Is it possible to control memory allocation for such case?To compare (but this is not a good comparison) the same snapshot
BoucyCastlerestores without additional memory.