Fixes also unwrapping for RSA keys, but we might need better solution.
I think the DerEncBigUint ensures the BN is unsigned and starts with zero byte, which is something we do not want when we import the BN as a modulus as you are using the modulus size to enforce the buffer size. Without stripping the null byte from the modulus, the operation requires 257 B buffer for signature and verification of this signature does not work.
The RSA wrapping is also using the RSAPrivateKey ASN1 directly, but from my reading of the PKCS#11 specification, we should use PrivateKeyInfo. This not obvious for RSA keys as everything needed is in the inner object, but EC keys needs the privateKeyAlgorithm field to learn what curve is used, if I read it right.
Keeping it as a draft as there is more quesitons to resolve before merging.
Fixes also unwrapping for RSA keys, but we might need better solution.
I think the
DerEncBigUintensures the BN is unsigned and starts with zero byte, which is something we do not want when we import the BN as a modulus as you are using the modulus size to enforce the buffer size. Without stripping the null byte from the modulus, the operation requires 257 B buffer for signature and verification of this signature does not work.The RSA wrapping is also using the
RSAPrivateKeyASN1 directly, but from my reading of the PKCS#11 specification, we should usePrivateKeyInfo. This not obvious for RSA keys as everything needed is in the inner object, but EC keys needs theprivateKeyAlgorithmfield to learn what curve is used, if I read it right.Keeping it as a draft as there is more quesitons to resolve before merging.