PKCS OAEP padding always returns: Pkcs11(ArgumentsBad)

[probablynachi]

This OAEP code complies but then throws the Pkcs11(ArgumentsBad) error for any parameters set. I am using softhsm2. Here is the code

use cryptoki::context::{CInitializeArgs, Pkcs11};
use cryptoki::mechanism::{Mechanism, MechanismType};
use cryptoki::mechanism::rsa::{PkcsMgfType, PkcsOaepParams, PkcsOaepSource};
use cryptoki::object::Attribute;
use cryptoki::session::UserType;
use std::error::Error;
use cryptoki::types::AuthPin;
const PKCS11_LIB_PATH: &str = "PATH";
const SLOT_ID: u64 = SLOT;
fn main() -> Result<(), Box<dyn Error>> {
    let context = Pkcs11::new(PKCS11_LIB_PATH)?;
    context.initialize(CInitializeArgs::OsThreads)?;
    let session = context.open_rw_session(SLOT_ID.try_into()?)?;
    session.login(UserType::User, Some(&AuthPin::new("PIN".into()))).expect("Invalid PIN");
    let pub_key_template = [Attribute::ModulusBits(2048.into())];
    let (publickey, _privkey) = session.generate_key_pair(&Mechanism::RsaPkcsKeyPairGen,
                                                       &pub_key_template, &[])?;
    let oaep=PkcsOaepParams::new(MechanismType::SHA1 ,PkcsMgfType::MGF1_SHA1, PkcsOaepSource::empty());
    let encrypt_mechanism: Mechanism = Mechanism::RsaPkcsOaep(oaep);

    session.encrypt(&encrypt_mechanism, publickey, b"Test")?;
    Ok(())
}

Which results in the error: Error: Pkcs11(ArgumentsBad). I tried the same code without using OAEP and it works as well, So i presume it is an issue with OAEP Padding. Thank you

[wiktor-k]

:thinking: It seems it should be supported in SoftHSM:

$ pkcs11-tool --modul /usr/lib64/pkcs11/libsofthsm2.so -M
Using slot 0 with a present token (0x63baf6de)
Supported mechanisms:
  AES-CBC, keySize={16,32}, encrypt, decrypt, wrap
...
  RSA-PKCS-KEY-PAIR-GEN, keySize={512,16384}, generate_key_pair
  RSA-PKCS-OAEP, keySize={512,16384}, encrypt, decrypt, wrap, unwrap // <-- here
  RSA-PKCS-PSS, keySize={512,16384}, sign, verify
...

If it's not a big problem could you file a PR with a test case showing the failure? Maybe then it'd be easier to spot the error/issue.

[probablynachi]

Yes i did recreate the same using python's pkcs11 library to see if it was a softhsm issue but I didn't face the error there, I will go ahead and try to create a test case showing the failure and file a PR as soon as i can. Thank you

[probablynachi]

I have created the PR and even added the softhsm file if needed along with it. Please get back if you need any information.