sfackler
commented
1 month ago I vaguely remember set_iv_length returning an error when used with ciphers that don't support IV length changes. If that's not the case, your proposed solution seems fine to me.
What a bizarre decision on the OpenSSL side of things...
I have been trying to use an IV of length 12 for encryption with AES 256 CCM but the result is as if I had used an IV of length 7. I suspect that this might be an issue for other ciphers that use an IV as well but I haven't tested.
This is observed using https://docs.rs/openssl/0.10.64/openssl/index.html (latest at time of writing) and OpenSSL 3.3.0.
The problem appears to be that the AES 256 CCM cipher defines the default IV length as 12, but OpenSSL still defaults to 7, causing the last 5 bytes in a 12 byte IV to be ignored.
Minimal reproducible example to show that 7 and 12 byte IV gives the same result:
Details
```rust fn main() { let cipher = openssl::symm::Cipher::aes_256_ccm(); let shared_key = { let peer_public_key = openssl::pkey::PKey::public_key_from_raw_bytes( &[ 96, 12, 76, 143, 122, 170, 174, 93, 156, 154, 227, 185, 173, 46, 41, 35, 235, 2, 22, 165, 46, 125, 79, 218, 86, 184, 14, 121, 206, 138, 227, 84, ], openssl::pkey::Id::X25519, ) .unwrap(); let private = openssl::pkey::PKey::private_key_from_raw_bytes( &[ 19, 163, 196, 101, 143, 222, 15, 123, 7, 149, 252, 132, 136, 99, 29, 231, 1, 16, 107, 167, 59, 16, 49, 186, 3, 145, 119, 78, 56, 25, 6, 60, ], openssl::pkey::Id::X25519, ) .unwrap(); let mut deriver = openssl::derive::Deriver::new(&private).unwrap(); deriver.set_peer(&peer_public_key).unwrap(); deriver.derive_to_vec().unwrap() }; let plain_text = &[ 3, 2, 9, 16, 19, 193, 87, 25, 215, 46, 127, 232, 223, 212, 18, 211, ]; let iv_7 = [155, 127, 74, 111, 232, 50, 255]; let iv_12 = [155, 127, 74, 111, 232, 50, 255, 59, 59, 70, 43, 169]; assert_eq!(iv_7, iv_12[..7]); let iv_7 = Some(iv_7.as_slice()); let iv_12 = Some(iv_12.as_slice()); let mut tag_7 = [0; 12]; let encrypted_7 = openssl::symm::encrypt_aead(cipher, &shared_key, iv_7, &[], plain_text, &mut tag_7) .unwrap(); let mut tag_12 = [0; 12]; let encrypted_12 = openssl::symm::encrypt_aead(cipher, &shared_key, iv_12, &[], plain_text, &mut tag_12) .unwrap(); assert_eq!(encrypted_7, encrypted_12); assert_eq!(tag_7, tag_12); } ```
From my testing, removing the inner length comparison of
https://github.com/sfackler/rust-openssl/blob/5095d7da4e26ec06ff29948068430379dac8f5f1/openssl/src/symm.rs#L629-L633
and unconditionally setting the IV length fixes the issue.
I think that this might be an appropriate solution for the problem and would be happy to submit a PR for it!