search

smuellerDD / acvpproxy

ACVP Proxy for accessing the NIST ACVP server for testing cryptographic implementations
https://www.chronox.de/acvpproxy
Other
16 stars 10 forks source link

Malloc Error #12

Closed celic closed 5 years ago

celic commented 5 years ago

I ended up hitting this error again. Here is the message reported.

acvp-proxy(4014,0x700010c4b000) malloc: *** error for object 0x7ff93941af20: pointer being freed was not allocated
acvp-proxy(4014,0x700010c4b000) malloc: *** set a breakpoint in malloc_error_break to debug

Do you know how to prevent it? Or what causes it?

smuellerDD commented 5 years ago

Am Mittwoch, 22. Mai 2019, 17:44:37 CEST schrieb Chris Celi:

Hi Chris,

I ended up hitting this error again. Here is the message reported.


acvp-proxy(4014,0x700010c4b000) malloc: *** error for object 0x7ff93941af20:
pointer being freed was not allocated acvp-proxy(4014,0x700010c4b000)
malloc: *** set a breakpoint in malloc_error_break to debug ```

Do you know how to prevent it? Or what causes it?

Can you please send me the output of

otool -L acvp-proxy

The reason I am asking is that assume it is related to the OpenSSL locking issue.

How often do you see that issue?

Do you see the issue when you use the command line -v -v -v (which disables threading and thus the OpenSSL problem is not hit)?

Ciao Stephan

celic commented 5 years ago

Unfortunately to test the bottleneck, I need the threading enabled, so I have not tested with -v -v -v.

Here is the output of otool -L acvp-proxy

acvp-proxy:
    /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1570.15.0)
    /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
    /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1570.15.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
celic commented 5 years ago

Could you please clarify what OpenSSL does here? I don't need an IUT to make these requests. Is OpenSSL providing some interfaces for the HTTPS connection? Or is it something I could disable entirely?

smuellerDD commented 5 years ago

Am Mittwoch, 22. Mai 2019, 18:02:11 CEST schrieb Chris Celi:

Hi Chris,

Unfortunately to test the bottleneck, I need the threading enabled, so I have not tested with -v -v -v.

Here is the output of otool -L acvp-proxy

Let me test some more here.

Ciao Stephan

smuellerDD commented 5 years ago

Am Mittwoch, 22. Mai 2019, 18:03:22 CEST schrieb Chris Celi:

Hi Chris,

Could you please clarify what OpenSSL does here? I don't need an IUT to make these requests. Is OpenSSL providing some interfaces for the HTTPS connection? Or is it something I could disable entirely?

The issue is that up to and including OpenSSL 1.0.2, OpenSSL required the caller to provide locking primitives. If these locking primitives are not registered with OpenSSL, it will randomly crash if you do use it in multithreaded environments.

But starting with OpenSSL 1.1.x, OpenSSL itself is fully thread-safe.

So, for OpenSSL 1.0.x, the code lib/openssl_thread_support.c must register the callbacks.

Note, even the acvp-proxy does not make use of OpenSSL, libcurl does. And libcurl does not make any precautions for OpenSSL thread-safety either. Thus, the proxy must ensure this.

Ciao Stephan

celic commented 5 years ago

Could I update locally to 1.1.x so that libcurl uses a properly threaded version of OpenSSL? If it helps, I am on OSX (which I think just symlinks OpenSSL to LibreSSL; I'm not familiar enough to know the differences).

celic commented 5 years ago

I installed OpenSSL 1.1.x locally and modified the Makefile to point to those libraries (which I am hoping is what libcurl is looking for). I'll test with that and see if that fixes the error for me.

smuellerDD commented 5 years ago

Am Mittwoch, 22. Mai 2019, 18:18:06 CEST schrieb Chris Celi:

Hi Chris,

Could I update locally to 1.1.x so that libcurl uses a properly threaded version of OpenSSL? If it helps, I am on OSX (which I think just symlinks OpenSSL to LibreSSL; I'm not familiar enough to know the differences).

Ahhh, that may be certainly an issue. LibreSSL is a fork of OpenSSL. So at least they used to have the issue too. I have no idea when or whether that issue was solved. After looking into the upstream code, it seems that this issue is still present.

With that said, it is very likely that the OpenSSL threading code is not compiled. See in lib/openssl_thread_support.c: The code is only compiled if it finds the OpenSSL version < 1.1.0:

if OPENSSL_VERSION_NUMBER < 0x10100000L

If you use libreSSL, that macro is not applicable and the code is not compiled.

One recommendation, can you either turn the aforementioned #if into an #if 1 or use the macro below and recompile the code?

if ((OPENSSL_VERSION_NUMBER < 0x10100000L) || defined

LIBRESSL_VERSION_NUMBER)

Or you use OpenSSL 1.1.x

Ciao Stephan

celic commented 5 years ago

I have no issues since updating to OpenSSL 1.1.x :^)

Thank you for the assistance.

smuellerDD commented 5 years ago

Am Mittwoch, 22. Mai 2019, 20:28:40 CEST schrieb Chris Celi:

Hi Chris,

I have no issues since updating to OpenSSL 1.1.x :^)

Thank you for the assistance.

Oh, my. I will add the mentioned ifdef for LibreSSL as well.

Thanks a lot for reporting it. Just like I do: keep the reports coming :-)

Ciao Stephan

celic commented 5 years ago

It feels nice to switch the roles every once in a while ;)

smuellerDD commented 5 years ago

Short followup for the records. I managed to trigger the issue myself. The following backtrace is visible:

So, the issue is definitely somewhere in the OpenSSL code. And it seems that the mutex code I use does not seem to fully solve the issue. And my default OpenSSL version 1.0.2r.

smuellerDD commented 5 years ago

Confirming that it is an OpenSSL bug:

When using the ACVP Proxy code on MacOS and after converting curl to use Apple Secure Transport and applying the patch given in https://github.com/curl/curl/pull/3933 to curl no malloc issues or other random crashes were observed.

Note, you can easily obtain curl linked with Apple Secure Transport by using Homebrew: brew install curl will give you a curl version linked with Apple Secure Transport. Just make sure you wait until the bug fix mentioned above is applied to Homebrew. See also https://github.com/Homebrew/brew/issues/6170 .

Using ACVP Proxy on Linux with curl linked to OpenSSL 1.1.x like on Fedora 29 does not exhibited such random crashes. This is another confirmation that OpenSSL <= 1.0.2 has still issues with threading even though threading callbacks as defined in https://linux.die.net/man/3/crypto_lock are set.