HTTPS requests to CA with AES-CBC failing with java.lang.ArrayIndexOutOfBoundsException: javax.crypto.ShortBufferException: Need at least X bytes of space in output buffer

[SilleBille]

This issue was migrated from Pagure Issue #3099.Originally filed by msqr on 2019-04-28

I am testing a new deployment of Dogtag 10.6.9 on Fedora 29. When I run some tests that attempt to submit a CSR using an agent user with a client certificate, I am getting an SSL error that I've tracked down on the Dogtag side of the SSL connection. The SSL handshake and certificate exchange all appear to succeed in Dogtag, but then fail when responding to the client. The error looks like this (in the SSL debug logs on the Dogtag server):

https-jsse-nio-8443-exec-12, READ: TLSv1.2 Handshake, length = 96
check handshake state: finished[20]
update handshake state: finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
*** Finished
verify_data:  { 159, 80, 213, 165, 89, 51, 182, 5, 11, 2, 77, 159 }
***
update handshake state: change_cipher_spec
upcoming handshake states: server finished[20]
https-jsse-nio-8443-exec-12, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data:  { 94, 190, 41, 194, 207, 95, 63, 156, 11, 15, 11, 26 }
***
update handshake state: finished[20]
https-jsse-nio-8443-exec-12, WRITE: TLSv1.2 Handshake, length = 96
%% Cached server session: [Session-34, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
https-jsse-nio-8443-exec-11, fatal error: 80: problem unwrapping net record
java.lang.ArrayIndexOutOfBoundsException: javax.crypto.ShortBufferException: Need at least 1584 bytes of space in output buffer
%% Invalidated:  [Session-34, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
https-jsse-nio-8443-exec-11, SEND TLSv1.2 ALERT:  fatal, description = internal_error
https-jsse-nio-8443-exec-11, WRITE: TLSv1.2 Alert, length = 80

I found out that the following ciphers all produced the same stye error:

• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
• TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
• TLS_DHE_RSA_WITH_AES_256_CBC_SHA
• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
• TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
• TLS_DHE_RSA_WITH_AES_128_CBC_SHA

Until I reached the first cipher that worked: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384. It looks like the AES-CBC ciphers are all failing, but AES-GCM is OK.

[SilleBille]

Posted by ftweedal on 2019-04-28:

Is the full traceback available anywhere, from the debug log or the system journal?

[SilleBille]

Posted by msqr on 2019-04-28:

I couldn't find a stacktrace anywhere, but I turned on JVM debugging and captured a stacktrace in Eclipse, which I'm attaching as a screen shot. The exception comes from the org.mozilla.jss.provider.javax.crypto.JSSCipherSpi.AES.bufferCrypt() method, line 759.