OpenSSL "Verification failure" with jsrsasign Signature

[fgsalomon]

Hello,

This issue is related to the last comments in #217 but since the issue is very old I've opened a new one.

I have a RSA key pair of 4096 bits and I want to sign binary data with jsrsasign and then verify the signature in the server with a different library (the backend uses a different programming language).

In my tests I can create the signature with jsrsasign and even verify it against the original data. However, if I use the OpenSSL CLI to verify the signature it always return a "Verification failure".

I've created a JSFiddle for testing here. Here, after setting the keys you can select a file and generate a signature and verify it with jsrsasign. The signature will be downloaded as signature.blob.

This is how I verify the signature with OpenSSL:

openssl dgst -verify public-key.pem -keyform PEM -sha256 -signature signature.blob --binary filetosign.bin
Verification failure
00D03DDC01000000:error:02000068:rsa routines:ossl_rsa_verify:bad signature:crypto/rsa/rsa_sign.c:430:
00D03DDC01000000:error:1C880004:Provider routines:rsa_verify:RSA lib:providers/implementations/signature/rsa_sig.c:788:

The JSFiddle also allows to upload a signature file and verify it against the upload file. If I use the jsrsasign generated signature it works OK but it I use an OpenSSL generated signature it fails:

This is how I generate the signature with OpenSSL:

openssl dgst -sha256 -sign private-key.pem -out openssl-signature.bin -binary filetosign.bin

I'm using OpenSSL 3.1.4 on Mac OS 14.2.1 and the latest version of jsrsasign.

Does anyone know what I'm doing wrong?

Thank you.

[kjur]

To sign a binary data, signHex shall be used instead of signString like this:

var rs = require('jsrsasign');
var rsu = require('jsrsasign-util');

var binFile = "filetosign.bin";
var prvFile = "public-key.pem";
var outFile = "signature.blob";
var hashAlg = "SHA256withRSA";

var prvPEM = rsu.readFile(prvFile);
var prv = rs.KEYUTIL.getKey(prvPEM);

var hexFile = rsu.readFileHexByBin(binFile);

var sig = new rs.KJUR.crypto.Signature({alg: hashAlg});
sig.init(prv);
var sigHex = sig.signHex(hexFile);
rsu.saveFileBinByHex(outFile, sigHex);

This will work fine for your data.

Here is my result:

$ od -c bbb.bin
0000000  001 002 003 004 005                                            
0000005
$ openssl dgst -verify a.pub -keyform PEM -sha256 -signature bbb.sig --binary bbb.bin
Verified OK