AES encryption is incompatible with java AES decryption - always gives BadPaddingException

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Use as3crypto AES to encrypt anything
2. try and decrypt it using javax.crypo 
3. You will always get: javax.crypto.BadPaddingException: Given final block not 
properly padded

Flash code snippets:

public function TestEncryption()
    {
    var encryptedBytes:ByteArray;   
    encryptedBytes = encrypt("1234567890123456");
    }
public static var KEY:ByteArray = 
Hex.toArray("faf579668b4bb03be0732eb0a42a7ebe");

public static function encrypt(data:String):ByteArray {
    var AES:AESKey = new AESKey(KEY);
    var dataByteArray:ByteArray =new ByteArray();
    trace("outgoing messsage [" + data + "]");
    dataByteArray.writeUTFBytes(data);
    trace("before encryption [" + Hex.fromArray(dataByteArray, true) + "]");
    trace("before encryption [" + Hex.fromArray(dataByteArray, false) + "]");

    var blocks:uint = dataByteArray.length / 16 
    // Pad out the array to exactly 16 byte blocks.
    var remainder:uint = dataByteArray.length % 16
    if ( remainder > 0)  {
    blocks ++;
    dataByteArray.length = 16*(blocks); // this right fills with \00
    }

    trace("before encryption after padding [" + Hex.fromArray(dataByteArray, true) + "]");
    trace("before encryption after padding [" + Hex.fromArray(dataByteArray, false) + "]");

    for (var block:uint=0; block < blocks; block++) {
    AES.encrypt(dataByteArray, block*16);
    }

    trace("after encryption  [" + Hex.fromArray(dataByteArray, true) + "]");
    trace("after encryption  [" + Hex.fromArray(dataByteArray, false) + "]");

    return dataByteArray;
} // encrypt()

This produces:
outgoing messsage [1234567890123456]
before encryption [31:32:33:34:35:36:37:38:39:30:31:32:33:34:35:36]
before encryption [31323334353637383930313233343536]
before encryption after padding 
[31:32:33:34:35:36:37:38:39:30:31:32:33:34:35:36]
before encryption after padding [31323334353637383930313233343536]
after encryption  [12:67:3b:79:94:50:4a:71:04:6c:86:58:03:15:5c:e4]
after encryption  [12673b7994504a71046c865803155ce4]

So the encrpted string is (hex): 12673b7994504a71046c865803155ce4

Java code snippets:

byte[] messageBytes = CryptUtils.hexToBytes(message);
String result;

public static void main(String[] args) throws NoSuchAlgorithmException, 
NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, 
BadPaddingException {

    byte[] messageBytes = CryptUtils.hexToBytes("12673b7994504a71046c865803155ce4");
    String strKeyHex="faf579668b4bb03be0732eb0a42a7ebe";
    byte[] raw = CryptUtils.hexToBytes(strKeyHex);
    KeyGenerator kgen = KeyGenerator.getInstance("AES");
    kgen.init(128); // 192 and 256 bits may not be available
    SecretKeySpec skeySpec = new SecretKeySpec(raw, "AES");
    Cipher cipher = Cipher.getInstance("AES");
    cipher.init(Cipher.DECRYPT_MODE, skeySpec);

    byte[] decrypted = cipher.doFinal(messageBytes);
}

public static byte[] hexToBytes(String hex) {
    byte[] bts = new byte[hex.length() / 2];
    for (int i = 0; i < bts.length; i++) {
       bts[i] = (byte) Integer.parseInt(hex.substring(2 * i, 2 * i + 2), 16);
    }
    return bts;
}

This always generates:
Exception in thread "main" javax.crypto.BadPaddingException: Given final block 
not properly padded
    at com.sun.crypto.provider.SunJCE_f.b(DashoA13*..)
    at com.sun.crypto.provider.SunJCE_f.b(DashoA13*..)
    at com.sun.crypto.provider.AESCipher.engineDoFinal(DashoA13*..)
    at javax.crypto.Cipher.doFinal(DashoA13*..)
    at com.skillkash.ge.api.TestEncrypt.decryptBytes(TestEncrypt.java:53)
    at com.skillkash.ge.api.TestEncrypt.main(TestEncrypt.java:32)algo: AES

Original issue reported on code.google.com by hob...@gmail.com on 9 Dec 2010 at 8:11

[GoogleCodeExporter]

Ive double checked tht hesToBytes works, by dumping the resultant byte buffer 
as decimal integers, and manually checking each.

Original comment by hob...@gmail.com on 9 Dec 2010 at 9:20

[GoogleCodeExporter]

Interestingly, if you ask javax crypto to encrpt the same string using the same 
key, it comes up with an ecrpyted string which is much longer.

Encryption on the java side, and decryption on the flash side works.

Original comment by hob...@gmail.com on 9 Dec 2010 at 9:22

[GoogleCodeExporter]

OK, I have reversed engineered the example program, and realised I have been 
using the library completely wrongly for a long time.  I was manually trying to 
add padding, to avoid null pointer errors, but there is a level above AES which 
does this for you.

Also, to use this library, you have to become something of an expert in the 
undocumented paramers, such as ebc/cbc.  Note: cbc needs a second key, called 
an initialisation vector (IV).  So ebc is much simipler (but presumaly less 
secure).

However, I am not 100% sure this will work all the time - as Im not sure yet 
how to account for the fact that one side uses 16 byte padding, and the other 8 
byte.

This is how you are supposed to do it (flash side):

public static var KEY:ByteArray = 
Hex.toArray("faf579668b4bb03be0732eb0a42a7ebe");
public static function encrypt2(data:String):ByteArray {
    var dataByteArray:ByteArray =new ByteArray();
    dataByteArray.writeUTFBytes(data);
    trace("before encryption [" + Hex.fromArray(dataByteArray, true) + "]");
    trace("before encryption [" + Hex.fromArray(dataByteArray, false) + "]");
    var pad:IPad = new PKCS5;
    var name:String = "aes-ecb";
    var mode:ICipher = Crypto.getCipher(name, KEY, pad);
    pad.setBlockSize(mode.getBlockSize());
    mode.encrypt(dataByteArray);
    trace("after encryption  [" + Hex.fromArray(dataByteArray, true) + "]");
    trace("after encryption  [" + Hex.fromArray(dataByteArray, false) + "]");
    return dataByteArray;
} // encrypt()

java side (not sure if this is how you are supposed to do it, but works):

static String decryptBytes(String message) throws NoSuchAlgorithmException, 
NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, 
BadPaddingException{

    byte[] messageBytes = CryptUtils.hexToBytes(message);
    for (int i=0; i<messageBytes.length; i++) {
       System.out.println("Byte[" + i + "]" + (0x000000FF & (int) messageBytes[i]) );
    }

    String result;
    String strKeyHex="faf579668b4bb03be0732eb0a42a7ebe";
    byte[] raw = CryptUtils.hexToBytes(strKeyHex);      

    KeyGenerator kgen = KeyGenerator.getInstance("AES");
    kgen.init(128); // 192 and 256 bits may not be available
    SecretKeySpec skeySpec = new SecretKeySpec(raw, "AES");

    Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
    cipher.init(Cipher.DECRYPT_MODE, skeySpec);
    byte[] decrypted = cipher.doFinal(messageBytes);
    result = new String(decrypted);
    return result;
}

Original comment by hob...@gmail.com on 10 Dec 2010 at 11:38