Inconsisten length result of RSA with PKCS1 padding

[rickyazhari]

hi, i develop RSA PKCS1padding to encrypt my password text using pod Universal-openssl. The problem is when i call RSA_public_encrypt method the result of " (unsigned char *) to " length is inconsistent. In my case the right length value is 128, but sometimes the process return 0 length or any number. This is my sample code :

BIGNUM *xponent = BN_new();
    BIGNUM *modulus = BN_new();
    BN_hex2bn(&xponent,xponentInHex);
    BN_hex2bn(&modulus,modInHex);

    RSA *rsa = RSA_new();
    rsa->e = xponent;
    rsa->n = modulus;
    char encoded[1024] = {0};

    RSA_public_encrypt(
        (int)strlen(charString),// from len
        (const unsigned char *)charString, // from
        (unsigned char *)encoded, // to
        rsa,
        RSA_PKCS1_PADDING
    );

    RSA_free(rsa);
NSLog(@"%lu", strlen(encoded));

if any have issue about my implement. please let me know. thanks

[krzyzanowskim]

Your encoded array is 1024 bytes long, and it's an array of bytes, you can't treat it with strlen() as strlen is intended for "strings", hence if find 0 bytes, it will stop counting. The encoded length depends on your key length, and is < RSA_size(rsa) long.

see documentation: https://www.openssl.org/docs/manmaster/man3/RSA_public_encrypt.html

flen must be less than RSA_size(rsa) - 11 for the PKCS #1 v1.5 based padding modes, less than RSA_size(rsa) - 41 for RSA_PKCS1_OAEP_PADDING and exactly RSA_size(rsa) for RSA_NO_PADDING. The random number generator must be seeded prior to calling RSA_public_encrypt().

RSA_public_encrypt() returns the size of the encrypted data (i.e., RSA_size(rsa)). and this is the value u should use to get length of the encrypted data.

PS. I don't think this is the right place to ask about OpenSSL itself :) you may wan't to find OpenSSL forum and ask there.