libtom / libtomcrypt

LibTomCrypt is a fairly comprehensive, modular and portable cryptographic toolkit that provides developers with a vast array of well known published block ciphers, one-way hash functions, chaining modes, pseudo-random number generators, public key cryptography and a plethora of other routines.
https://www.libtom.net
Other
1.57k stars 458 forks source link

pkg-config file should preserve CFLAGS #671

Open deldotbrain opened 3 weeks ago

deldotbrain commented 3 weeks ago

Prerequisites

Description

I opened nixos/nixpkgs#346707 against nixpkgs while packaging an application that used its shared libtomcrypt library. Before I hack around the issue in nixpkgs, I thought I should report it here.

The pkg-config file installed by libtomcrypt doesn't capture the CFLAGS that LTC was built with. Applications using pkg-config to link against a shared LTC library will not automatically build with the same CFLAGS as LTC, which can cause problems. For some flags (e.g LTM_DESC in the application I was packaging), that's a nuisance. (Edit: I see that since the release of 1.18.2, a change was made so the LTM_DESC flag is captured in the pkg-config file)

However, for the LTC_PTHREAD flag specifically, it can cause a major bug. Since LTC_PTHREAD changes the size of struct prng_state, building LTC with it enabled (as nixpkgs does), then using pkg-config to build an application against it causes e.g. rng_make_prng() to overwrite glibc's malloc metadata and crash the program.

Steps to Reproduce

$ make CFLAGS="-DUSE_LTM -DLTM_DESC -DLTC_PTHREAD" EXTRALIBS="-ltommath" -f makefile.shared install

$ cat <<EOF >test.c
#include <stdio.h>
#include <tomcrypt.h>

int main(int argc, char **argv) {
  // Initialize the yarrow PRNG, including mutexes for which no space was
  // allocated:
  int yarrow_wprng = register_prng(&yarrow_desc);
  prng_state* yarrow_state = malloc(sizeof(prng_state));
  if (rng_make_prng(128, yarrow_wprng, yarrow_state, NULL) == -1) {
    fprintf(stderr, "error initializing prng");
    return 1;
  }

  // LTC has now accidentally corrupted glibc's malloc metadata for
  // yarrow_state. Doing another allocation will detect it.
  malloc(1);

  // If prng_state were stack-allocated instead, SSP would detect a
  // stack smashing attempt when main() returns.
  return 0;
}
EOF

$ gcc $(pkg-config --cflags --libs) test.c && ./a.out

The test program will crash and burn with a failed assertion from glibc.

Version

libtomcrypt 1.18.2

GCC 13.3.0 with glibc 2.39-52 on NixOS unstable (24.11)

levitte commented 3 weeks ago

I've thought of this discrepancy in the last few weeks, but haven't had the peace of mind to figure out a solution.

I'm not at all convinced that the user CFLAGS should be the same as the build CFLAGS. it shouldn't be necessary... but perhaps we'll have to in some sort of interim.