`c4test` frequently crashes with "malloc: Region cookie corrupted" when calling `EVP_MD_CTX_new()`

[mallman]

I'm exploring the suitability of using the hatrack portion of libcon4m for my own project(s). I'm running into some problems there, but before I go further into analyzing my own code, I want to report a crash I'm seeing in c4test.

When I run c4test it frequently—but not always—crashes in the sha_init() function when it calls EVP_MD_CTX_new(). Specifically, I'm referring to line 55 in

https://github.com/crashappsec/libcon4m/blob/9fdd12cccd636b8ec5a8c1a2110997832ddced4a/src/con4m/crypto/sha.c#L33-L59

I've built libcon4m on an arm64 Mac. Unfortunately (or not), macOS does not (seem to) come with a system -lcrypto. I'm using OpenSSL 3.2.1 from MacPorts. If you've built and run c4test on a Mac, which crypto library are you linking against and how did you install it? And are you able to run c4test repeatedly with no malloc errors?

[viega]

Sorry, just saw this. I'm on an arm mac too, but linking to an OpenSSL I built. I can share the .a file directly if that's helpful?

[mallman]

It turns out I could not reproduce this problem in Apple's Terminal app. I can reproduce it reliably in VSCode's terminal. Comparing environment variables, I found that VSCode's terminal's environment defines a suspicious-looking MallocNanoZone=0. I found that setting MallocNanoZone=0 in the Terminal app environment caused c4test to crash, just like it does in VSCode's terminal. Give it a try.

I could not find much information on the MallocNanoZone environment variable, but it seems to be used to help debug memory allocation errors. I wonder if this indicates some kind of malloc bug in OpenSSL? I'm tempted to file an issue with that project, or debug this problem myself. I don't suppose you might have any insight yourself?

[viega]

Ah. That environment variable forcibly changes the system library's memory allocator. I don't know how it does it, but I would suspect it essentially does muck with redirecting function calls. That would be fine for using hatrack w/o con4m, but since con4m is basically adding a garbage collection layer, it's not too much of a surprise there'd be some conflict, depending on how openssl determines how to redirect malloc.

If it's an issue to disable nanomalloc, I can try to spend some time on it over the weekend, as I'm sure I could accomidate that but I don't see why one would need to use nanomalloc either :)

[mallman]

Hi @viega. The issue is just running c4test in the VSCode terminal. At this point it sounds like I won't be using the garbage collection facility, so I'll just close this issue now.

Thanks.