hashAndSign() applies PKCS#1 padding twice

[GoogleCodeExporter]

Problem:
I am having trouble with hashAndSign() form gdata.tlslite.utils.RSAKey. It 
appears to apply PKCS#1 padding twice, first in hashAndSign(), and then again 
in sign(). This produces an unexpected signature.

Workaround:
Avoid using hashAndSign(). Instead, use sha1() and sign() directly.

Version:
I am using gdata-2.0.13 with Python 2.5 on Mac OS X 10.5.

To reproduce:
Below is a series of commands and scripts to reproduce the problem.

$ cat msg
Lorem ipsum
$ openssl genrsa -out key.pem
Generating RSA private key, 512 bit long modulus
........++++++++++++
..++++++++++++
e is 65537 (0x10001)
$ cat msg | openssl dgst -sha1 -binary | openssl rsautl -inkey key.pem -sign | 
hexdump
0000000 2a 46 b2 25 67 bb 79 38 4a 12 cf e9 96 07 3b 1a
0000010 49 ca bf e2 29 9d f8 6c f7 7d 9a 59 eb 31 7c e4
0000020 a6 10 80 e3 37 03 d8 95 88 b3 a4 d9 46 05 99 c9
0000030 61 fb 5f c1 87 df 62 5e b3 87 8e e3 f8 24 6c b4
0000040
$ cat sign-ok.py 
#!/usr/bin/env python

from sys import stdout
from hashlib import sha1
from gdata.tlslite.utils import keyfactory, compat

def sign():
    f = open('msg')
    msg = f.read()
    f.close()

    f = open('key.pem')
    rsa_key = f.read()
    f.close()

    private_key = keyfactory.parsePrivateKey(rsa_key)
    digest = sha1(msg).digest()
    digest = compat.stringToBytes(digest)
    sig = private_key.sign(digest)

    stdout.write(compat.bytesToString(sig))

if __name__ == '__main__':
    sign()

$ ./sign-ok.py | hexdump
0000000 2a 46 b2 25 67 bb 79 38 4a 12 cf e9 96 07 3b 1a
0000010 49 ca bf e2 29 9d f8 6c f7 7d 9a 59 eb 31 7c e4
0000020 a6 10 80 e3 37 03 d8 95 88 b3 a4 d9 46 05 99 c9
0000030 61 fb 5f c1 87 df 62 5e b3 87 8e e3 f8 24 6c b4
0000040
$ cat sign-err.py 
#!/usr/bin/env python

from sys import stdout
from hashlib import sha1
from gdata.tlslite.utils import keyfactory, compat

def sign():
    f = open('msg')
    msg = f.read()
    f.close()

    f = open('key.pem')
    rsa_key = f.read()
    f.close()

    private_key = keyfactory.parsePrivateKey(rsa_key)
    sig = private_key.hashAndSign(msg)

    stdout.write(compat.bytesToString(sig))

if __name__ == '__main__':
    sign()

$ ./sign-err.py | hexdump
0000000 af 21 29 3e f9 d4 bd 9e 4d d1 40 6e 19 56 f8 ba
0000010 6e 3f a3 77 20 aa f6 52 cc 87 cd 8f 1d 60 33 46
0000020 78 4c 1f 80 c1 f9 58 7d 4e 68 77 86 5d 4c f5 e6
0000030 a5 80 58 2c bc 73 41 c0 f3 2b 1a 05 25 31 38 83
0000040
$ 

Original issue reported on code.google.com by dado.col...@gmail.com on 20 Nov 2010 at 3:19