libressl cannot decrypt its own smime encrypted files

[Supermathie]

We've discovered that LibreSSL isn't properly generating SMIME files. Specifically:

• LibreSSL 3.4.1 cannot decrypt an smime file generated by LibreSSL 3.4.1
• OpenSSL 1.1.1l cannot decrypt an smime file generated by LibreSSL 3.4.1
• LibreSSL 3.4.1 can decrypt an smime file generated by OpenSSL 1.1.1l

Note that in these commands below, openssl is 1.1.1l (on Arch) and libressl-openssl is 3.4.1 (on the same Arch system):

○ → openssl smime -decrypt -inkey test.key < testsecret-openssl.enc
hello world

○ → openssl smime -decrypt -inkey test.key < testsecret-libressl.enc
Error decrypting PKCS#7 structure
140596298573184:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:610:
#kv

○ → libressl-openssl smime -decrypt -inkey test.key < testsecret-openssl.enc
hello world

○ → libressl-openssl smime -decrypt -inkey test.key < testsecret-libressl.enc
Error decrypting PKCS#7 structure
139701602872192:error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt:evp/evp_enc.c:549:
gP

I don't know what exactly is going wrong, but it's happening on multiple computers.

This was first discovered on a co-worker's OS X system using the stock LibreSSL 2.8.3 - he was generating files that others could not decrypt. Upon investigation we found:

• LibreSSL 3.4.1 cannot decrypt an smime file generated by LibreSSL 2.8.3
• OpenSSL 1.1.1l cannot decrypt an smime file generated by LibreSSL 2.8.3
• LibreSSL 2.8.3 can decrypt an smime file generated by LibreSSL 3.4.1
• LibreSSL 2.8.3 can decrypt an smime file generated by LibreSSL 2.8.3

My demonstration files are:

test.key.gz (demo RSA key) test.crt.gz (public portion) testsecret-libressl.enc.gz (encrypted file generated by libressl) testsecret-openssl.enc.gz (encrypted file generated by openssl)

These files were generated on the Arch system mentioned above with:

openssl genrsa -out test.key 2048
openssl req -new -x509 -nodes -days 365 -key test.key -out test.crt
echo "hello world" > testsecret
openssl smime -encrypt -in testsecret -out testsecret-openssl.enc test.crt
libressl-openssl smime -encrypt -in testsecret -out testsecret-libressl.enc test.crt

[4a6f656c]

I am able to decrypt both of those files with both LibreSSL 3.5.0 (in development) and OpenSSL 1.1.1l:

$ eopenssl11 version
OpenSSL 1.1.1l  24 Aug 2021
$ eopenssl11 smime -decrypt -recip test.crt -inkey test.key < testsecret-libressl.enc  
hello world
$ openssl version
LibreSSL 3.5.0
$ openssl smime -decrypt -recip test.crt -inkey test.key < testsecret-libressl.enc     
hello world

Note that the -recip option is given to specify the certificate, otherwise it will result in the error you gave:

$ openssl smime -decrypt -inkey test.key < testsecret-libressl.enc                  
Error decrypting PKCS#7 structure
16157693055912:error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt:/usr/src/lib/libcrypto/evp/evp_enc.c:549:
$ eopenssl11 smime -decrypt -inkey test.key < testsecret-libressl.enc 
Error decrypting PKCS#7 structure
0:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:610:

I'm not aware of that changing between versions though.

[4a6f656c]

I'm going to close this due to there being no further follow up.