RSAKey verify error (PKCS#1 unpad: i=0, expected b[i]==[0,1,2])

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Have an SSL cert signed (issued) by DigiCert Global CA (see attached file)
2. Try to validate the SSL Cert
3. Get error: "PKCS#1 unpad: i=0, expected b[i]==[0,1,2], got b[i]=51"

At first I was getting errors about not being able to find the parent cert
for the SSL certificate I was trying to validate, so I had to add it
manually (the issuer is DigiCert Global CA -
http://www.digicert.com/CACerts/DigiCertGlobalCA.crt).

After that, I get an error when it's trying to verify the RSAKey associated
with that certificate.  The error occurs in the pkcs1unpad function. Does
anyone know what is going wrong?

Original issue reported on code.google.com by SilentCh...@gmail.com on 30 Sep 2009 at 5:39

Attachments:

• MozillaRootCertificates.as

[GoogleCodeExporter]

Perhaps this is caused by the same problem causing defect #27
Are you running 64bit flash?

Original comment by ge...@spacevs.com on 28 Oct 2009 at 11:31

[GoogleCodeExporter]

As far as I know I'm not running it in 64-bit mode.  This is happening on a mac
running Leopard.

Original comment by SilentCh...@gmail.com on 28 Oct 2009 at 11:59

[GoogleCodeExporter]

I'm having the same problem trying to use the verify function.  I'm running 32 
bit
debug flash on a windows machines, and I've implemented the fix for #27 just in 
case. 

I'm pretty sure that isn't the issue here.

Original comment by MEHender...@gmail.com on 20 Jan 2010 at 7:44

[GoogleCodeExporter]

I bashed my head against this for a while and figured out the problem I was 
getting:
I was generating the signature in php and writing it to a file, and I needed to
base_64 encode it first.

in php:
openssl_sign($text, $signature, $private_key);
$signature= base64_encode($signature);

then in flash:
var src:ByteArray =Base64.decodeToByteArray(signature);
var dst:ByteArray = new ByteArray();
verifier.verify(src, dst, src.length);

If you're using a pre generated certificate, there's probably a good chance 
that you
have a similar formatting problem.  Your signature should consist mainly of 
letters
and numbers :
UyaNH1l7O+V5kDGGWlGTapTVREXAHbgGDSiZ7gnSTVAetu43wH6yGa6Hzpb7BHq4sx9vPDHcHHtqnnpZ
BYS8KQ==

and not be something like this:
ÒMP¶î7À~²®Îûz¸³o<1Ü{jzY iY{;åy1ZQjÕDEÀ¸

I'm still working on getting everything to work properly, but I'm no longer 
getting
"PKCS#1 unpad: i=0, expected b[i]==[0,1,2], got b[i]=51" errors from 
RSAKey.verify

Original comment by MEHender...@gmail.com on 22 Jan 2010 at 1:57

[GoogleCodeExporter]

It may be worth checking out this 
http://ria101.wordpress.com/2010/06/04/as3crypto-rsa-padding-function-
returned-null-bug-of-death-fix/

Original comment by thedwill...@gmail.com on 4 Jun 2010 at 4:27

[GoogleCodeExporter]

In case someone else has the same problem, my test case was :

var data : ByteArray = new ByteArray;
for ( var k : int = 0 ; k < 25000 ; k++ )
    data.writeUnsignedInt( uint.MAX_VALUE * Math.random() ); 
var sha : SHA256 = new SHA256();
var hash : ByteArray = sha.hash( data );
var rsa : RSAKey = RSAKey.generate( 512, "0x10001" );
var signature : ByteArray = new ByteArray;
rsa.sign( hash, signature, hash.length );
var output : ByteArray = new ByteArray;
rsa.verify( signature, output, signature.length );

After a few hours trying to adjust the padding functions, I ran through the 
demo code and saw that the second argument of RSAKey.generate take a non 
standard hex string "10001 instead of "0x10001".

So i just changed :
var rsa : RSAKey = RSAKey.generate( 512, "0x10001" );
To :
var rsa : RSAKey = RSAKey.generate( 512, "10001" );
And everything worked perfectly.

I felt ashamed for a moment but finally I think the real problem is the missing 
documentation. I don't know if it will solve your problem but it can help 
someone else.

Original comment by raou...@gmail.com on 28 Jul 2010 at 8:57

[GoogleCodeExporter]

Adding some data to this issue. I've been taking a second shot at trying to 
figure it out, but I think the issue may be in the BigInteger class which is a 
little too complicated for me.

The cert I'm trying to use in my application is signed by DigiCert using a cert 
of their's with the subject:
"C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert High Assurance CA-3"

I've attached the PEM-encoded cert which I used to generate the block I needed 
to add to MozillaRootCertificates.as (DigiCertHighAssuranceCA-3.pem). The cert 
was exported from Firefox. To get the C code used in the parsing tool 
grabRootCAs.pl, I used the command:

> openssl x509 -in DigiCertHighAssuranceCA-3.pem -inform PEM -C

I then used the hex-encoded subject_name variable and performed the 
transformation done in grabRootCAs.pl to come up with a subject value of:

MGYxCzAJBgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdp
Y2VydC5jb20xJTAjBgNVBAMTHERpZ2lDZXJ0IEhpZ2ggQXNzdXJhbmNlIENBLTM=

The addPEMCertificate call is in the DigiCert CA-3.txt file.

Once this was complete, I tried connecting to a TLS socket which uses a cert 
signed using the certificate I just added. When the code is trying to call 
RSAKey.verify on the RSA key in the DigiCert CA-3 certificate, it fails:

PKCS#1 unpad: i=0, expected b[i]==1, got b[i]=5e
TLSEngine shutdown triggered by Error: Decrypt error - padding function 
returned null!

Original comment by SilentCh...@gmail.com on 16 Feb 2011 at 8:56

Attachments:

• [DigiCert CA-3.txt](https://storage.googleapis.com/google-code-attachments/as3crypto/issue-26/comment-7/DigiCert CA-3.txt)
• DigiCertHighAssuranceCA-3.pem

[GoogleCodeExporter]

I have exactly the same issue using an Entrust L1C cert, chained off the 
Entrust 2048 root.

Perhaps this is related to some kind of cert chaining issue?
Tracking though the code finally gets me to the failing unpad function where is 
appears that the result of the decrypt has resulted in a horribly broken 
BigInteger value.

Does anyone have any further insight?

Original comment by sweetlil...@gmail.com on 24 Feb 2011 at 1:42

[GoogleCodeExporter]

Hi,

I have found the issue and attached a diff that will fix this against the 1.3 
branch.

In short the issue is related to chained certs:

isSelfSigned() is broken. The pad function breaks (as it should) but this is 
incorrectly handled. This should fail gracefully allowing the code in 
isSigned() to chain to the next cert.

Original comment by sweetlil...@gmail.com on 28 Feb 2011 at 6:10

Attachments:

• as3crypto-certchain-fix.diff

[GoogleCodeExporter]

Just tried this and it looks like it works for my cert as well. Currently using 
the svn checkout version of the library, so had to make some small changes in 
the diff.

Thanks for posting this!

Original comment by SilentCh...@gmail.com on 28 Feb 2011 at 5:26