leycec
commented
3 years ago Relatedly, the LIEF_INSTALL_PYTHON CMake option declared by cmake/LIEFOptions.cmake silently reduces to a noop. It isn't actually used anywhere, but should be; ideally, enabling that option should instruct the api/python/CMakeLists.txt file to install the generated pyLIEF C extension into the site-packages/ subdirectory of the target Python interpreter.
Of course, setup.py exists – but that's really just a thin wrapper around cmake and ninja lacking most of the configurability of the top-level CMakeLists.txt file. Gentoo currently ignores LIEF's setup.py in lieu of directly invoking cmake and ninja, because patching setup.py to respect our CMake toolchain mostly seems like an exercise in futility. </sigh>
romainthomas
blshkv
Hi! I'm @leycec, the stalwart volunteer currently packaging the most recent stable release of LIEF (...so, LIEF 0.11.5) for Gentoo Linux. LIEF is a mandatory build-time dependency for CadQuery, which is what I'm really trying to package and which imports LIEF's Python API to dump symbols for Open Cascade.
Who cares! On with the ugly parade of bugs.
A Tale of Two Terrible Bugs
Okay. Let's admit that packaging LIEF hasn't gone well.
LIEF's CMake-based build system is surprisingly fragile, constantly falling down and refusing to get up in common use scenarios. Various options described by
cmake/LIEFOptions.cmakeappear to mostly be untested, as disabling those options induces fatal compilation errors – but only after ten to twenty minutes of successful compilation, which only complicates an already overcomplicated build system.The most egregious of these is the
-DLIEF_DEXoption governing DEX instrumentation. This option is thankfully enabled by default, because disabling this option destroys everything. What's mystifying is that the exact compilation error raised by disabling this option contextually changes based on which other CMake options are concurrently enabled or disabled.I'll list the two prominent errors I hit here, but there are likely more.
DEX Must Die
Let's first disable DEX and only DEX by passing
-DLIEF_DEX=OFF. Okay, we technically also disableccache, becauseccacheis ostensibly insane and we're trying to maximize sanity here:That's... not great. On the bright side, this failure erupts after only a minute or so of compilation; on the dark side, the endless litany of non-human-readable errors is somewhat less than ideal.
DEX, OAT, and VDEX Must Also Die
Let's next disable DEX, OAT, and VDEX altogether to circumvent the above error by passing
-DLIEF_DEX=OFF -DLIEF_OAT=OFF -DLIEF_VDEX=OFF.And... that actually works! The raccoons currently eating all of the tomatoes on our front deck rejoice. :raccoon: :tomato: :raccoon:
PE, ART, DEX, OAT, and VDEX Must All Die
Great. So, let's just go ahead and disable all format-specific instrumentation except for the two most popular POSIX-friendly formats (i.e., ELF and Macho-O) by passing
-DLIEF_PE=OFF -DLIEF_ART=OFF -DLIEF_DEX=OFF -DLIEF_OAT=OFF -DLIEF_VDEX=OFF.Because bugs breed like flies in the muggy Canadian heat, we now receive a completely different compilation error on a completely different compilation unit:
That's even worse. On the bright side, there's only a single prominent error; on the dark side, that error only erupts after ten to twenty minutes of compilation and (much more importantly) has no obvious relation to the underlying CMake options causing this error.
The superficial
"error: aggregate ‘std::ostringstream stream’ has incomplete type and cannot be defined"message just means there's a missing#include <sstring>inpyBinary.cpp, presumably due to some perfidious nesting of conditional includes. Debugging that error back to its root cause, however, took me the better part of several days.In reflection, that wasn't necessarily time well-spent.
Another Rotten Leaf Falls
Gentoo's LIEF package currently circumvents everything above by unconditionally enabling all format-specific CMake options (e.g., by internally forcing
-DLIEF_ELF=ON -DLIEF_PE=ON -DLIEF_MACHO=ON -DLIEF_ART=ON -DLIEF_DEX=ON -DLIEF_OAT=ON -DLIEF_VDEX=ON). That's really non-ideal for us, because the whole point of using Gentoo in the first place is fine-grained control over compilation. Still, working but non-configurable >>>>>>>> non-working but configurable.While debugging everything above, I slowly received the uncontrollable urge to break keyboards. It'd be great if these issues could be resolved to prevent any future hardship to keyboards.
As scarce volunteer time permits, it'd also be great if LIEF's continuous integration suite could be updated to test these problematic permutations of CMake options. Since I'll be packaging LIEF for the foreseeable future on Gentoo, I'm really hoping to avoid painful regressions in the future. Please help me help you.
And thanks for your tireless attention, LEIF developers. Maintaining a code behemoth of this magnitude cannot be easy. I salute your bravery and integrity in the face of ongoing development pain.