CMake build support?

[cblichmann]

Hey there, Is there any interest/ongoing work for having XED build with CMake? mbuild.py makes it a bit harder to integrate with other toolchains.

Cheers, Christian

[markcharney]

I do not use cmake so I cannot help you there. mbuild was an attempt to make a portable build system that relied only on python and worked on linux, windows and mac. While not a lot of other projects have adopted it, it has been successful for me since I've been able spend much less time dealing with the nuances of different build systems and external tools on the 3 O/Ses that I mentioned.

The way other projects build XED is to have their build system (whatever they are) invoke the python script to build XED just like running any tool required by a build system. Presumably you can do the same for your environment?

[cblichmann]

Sure, we all want to spend less time on build systems. In our environment it'll be difficult to call out to other build systems, so my best bet will likely be to modify mbuild to output a list of files along with their compile options and convert that to something I can use... Can I take your answer as "we accept PRs"? :)

[markcharney]

Can you explain why your build system cannot run a python script as part of the build process? Hard to fathom. Your proposal sounds much harder if not totally infeasible.

[cblichmann]

Think bazel.io on Google infrastructure (https://mike-bland.com/2012/10/01/tools.html#blaze-forge-srcfs-objfs details this a bit). The gist of it is that we don't usually allow code that is built with other build systems at Google. Whenever this has been done in the past using pre-built libs, it was a huge maintenance burden. No fault to the third_party code involved, just the way we compile things here :)

[markcharney]

Go ahead and send a PR. Once you see what is going on in the actual build, I doubt you'll make anything I could maintain going forward. Go for it though. I think an exception to your usual rules is more likely.

I am not religious about build systems. I look at mbuild as a handy python module that I use to build XED so I don't have to futz with the limitations of make, scons, nmake, etc. I just have to have one external tool other than the compiler tool chain. The existing python script invokes other XED python scripts to generate C code that is subsequently compiled. So I'm not sure where you'll want to draw the line (and why). Is it okay to invoke python for the generators but not for the overall build? It is just python scripts...

[michaeljclark]

Hi,

(apols in advance for the long comment - not religious about build systems either - but a pragmatic choice these days seems to be cmake - which I personally think is a bit clunky - but Xcode, Visual Studio, GNU make and ninja support along with the growing number of third party library descriptions makes it a rather attractive choice despite the clunky language)

cross-platform build systems

I'm working on a cross-platform C/C++ project that targets macOS, Windows, Android and GNU/Linux. I spent some time trying to see what build solution is in the most widespread use and I ended up choosing cmake, mostly out of pragmatism. I looked at cmake, gyp, bazel, scons and several others. The problem is not whether I like them, but rather whether are supported by common dependencies, or whether its quick to write one. So now i'm using cmake for about a dozen third-party dependencies, noting that it is becoming more and more common to find projects shipping with a CMakeLists.txt often as a primary build description.

xed

I'm currently using asmjit for x86 encoding. asmjit also uses cmake as its primary build description and its pragmatic way of avoiding cmake scripting is to check the generated files into the git repo.

I'm investigating xed as it supports encoding and decoding whereas asmjit is an assembler. A cmake build for xed will be necessary in one way or another if I decide to use xed. This could be done as an overlay calling mbuild or there might be some clean hybrid, that is maintainable and can coexists with mbuild.

vcpkg cmake overlays

Microsoft's Vcpkg contains cmake overlays for over one thousand libraries [1]: It seems like an obvious system to support so I'm also thinking about making a xed Vcpkg description which requires cmake. The other universal cross-platofrm build from source and dev package solution is conan, conan handles cmake projects with cmake builds using a cmake wrapper [2]. The reason I'm mentioning this is we could perhaps make a cmake wrapper to mbuild.

cmake strategies

It seems to me that a cmake build overlay for xed. There are a few alternatives:

• minimal cmake hack to check out and build mbuild and xed
  • run mbuild, locate xed headers and binary library files, and set INCLUDES and LIBRARY variables. Add a FindXed.cmake
  • Pros: most simple approach
  • Cons: end up needing to handle things we want to avoid like conditionals for build archive suffixes as the edge interface would be binary vs source. This means handing directories that won't be managed by the cmake build, We won't get source browsing in Visual Studio or Xcode (extremely useful for debugging). If it requires Xed to already be installed then it complicates CI as another build script to be run before cmake to avoid manual dependency installation.
• cmake mbuild hybrid
  • modify mbuild so that cmake can call the code to generate files and add command line options that return source lists much like pkg-config does.
  • Pros: maintains mbuild for generation, so can coexists with mbuild, gives us

cmake plan sketch

• cmake's execute_process command which we use to call mbuild to get lists of generated files
• cmake's add_custom_target command can be used to add generated targets dynamically
• cmake variables -DXED_OPTION_XXX=On can be used as alternatives to mbuild command line options. Yes. It's more clunky.
• preserve mbuild 100% preserved, but add additional options to be used by the cmake build such as command line interfaces to return target names, source lists for targets and to generate any generated files.
• we can avoid duplication by fetching source file lists from mbuild

My biggest issue is the separate mbuild repo as a dependency. cmake and python can be assumed to be installed. mbuild is essentially a passive dependency in this scenario and ideally could be included with xed. I guess we just use python $(PROJECT_SOURCE_DIR)/../mbuild.py and have both projects as submodules.

With this approach, the cmake would mostly be script using execute_process to fetch target list, and foreach loop calling execute_process to get source listings and dynamically add targets using add_custom_target command at configure time with build description that calls mbuild to generate the dynamically generated files at compile time. Here is psuedo-code (not cmake syntax)

• TARGET_LIST = $(execute_process mbuild targets)
• foreach (TARGET in TARGET_LIST)
  • TARGET_TYPE = $(execute_process mbuild target-type $TARGET) # 'library' or 'executable'
  • TARGET_SOURCE = $(execute_process mbuild source $TARGET)
  • TARGET_DYN_SOURCE = $(execute_process mbuild dyn-source $TARGET)
  • add_add_custom_target (with mbuild generate $GENERATED_FILE), add_library or add_executable

cmake scripts need to split configure time and compile time actions evaluating target names and their source listings would be done at configure time (invoking mbuild), but add_custom_target would have the generation called at build time. mbuild doesn't seem to have a separate configure step. is that correct?

action-items

None at present. not promising, just sharing some ideas on how it could be done so that cmake support could coexist with mbuild. Finding out how hard it is go get mbuild to get files lists, determine which ones are dynamic and how to call the generator for individual files, would be the next step if/when we get time to move forward on this. Don't want to block anyone by promising to do any of this. I guess we have a sketch/proposal for a fuzzy line between mbuild/cmake.

but I will really really like this if someone does this.

_

[1] https://github.com/microsoft/vcpkg/tree/master/ports [2] https://github.com/conan-io/cmake-conan

[mrexodia]

kits/xed-install-base-xxx/lib/cmake/XED/XEDConfig.cmake

set(XED_LIBRARY_DIR "${CMAKE_CURRENT_LIST_DIR}/../../../lib")
set(XED_LIBRARIES "${XED_LIBRARY_DIR}/xed${CMAKE_STATIC_LIBRARY_SUFFIX}" "${XED_LIBRARY_DIR}/xed-ild${CMAKE_STATIC_LIBRARY_SUFFIX}")
set(XED_INCLUDE_DIRS "${CMAKE_CURRENT_LIST_DIR}/../../../include")

# https://stackoverflow.com/a/48397346
add_library(XED::Main STATIC IMPORTED)
set_target_properties(XED::Main PROPERTIES
    IMPORTED_LOCATION "${XED_LIBRARY_DIR}/xed${CMAKE_STATIC_LIBRARY_SUFFIX}"
)

add_library(XED::ILD STATIC IMPORTED)
set_target_properties(XED::ILD PROPERTIES
    IMPORTED_LOCATION "${XED_LIBRARY_DIR}/xed-ild${CMAKE_STATIC_LIBRARY_SUFFIX}"
)

add_library(XED::XED INTERFACE IMPORTED)
set_target_properties(XED::XED PROPERTIES
    INTERFACE_INCLUDE_DIRECTORIES "${XED_INCLUDE_DIRS}"
    INTERFACE_LINK_LIBRARIES "XED::Main;XED::ILD"
)

Usage from your CMakeLists.txt:

find_package(XED REQUIRED)
target_link_libraries(MyProject PRIVATE XED::XED)
``

Add the absolute path to ``kits/xed-install-base-xxx` to your `CMAKE_PREFIX_PATH` so CMake can find this package. I built this based on https://www.youtube.com/watch?v=eC9-iRN2b04 and it should be modern CMake.

[SaifRushdHadad]

Has anyone made progress on this?

Cross compilation of this project from Debian to Windows is near impossible due to the limitations of the mbuild build system.

[pgoodman]

Microsoft's Vcpkg contains cmake overlays for over one thousand libraries [1]: It seems like an obvious system to support so I'm also thinking about making a xed Vcpkg description which requires cmake. The other universal cross-platofrm build from source and dev package solution is conan, conan handles cmake projects with cmake builds using a cmake wrapper [2]. The reason I'm mentioning this is we could perhaps make a cmake wrapper to mbuild.

We have a vcpkg XED overlay here: https://github.com/lifting-bits/cxx-common/tree/master/ports/xed

[rileysjc]

I've only tested on Win10+ but it handles the various WIN caveats. CMakeLists.txt XEDConfig.cmake.in.txt

[rileysjc]

Has anyone made progress on this?

Cross compilation of this project from Debian to Windows is near impossible due to the limitations of the mbuild build system.

Intel's mbuild is likely sufficient for Intel's needs but MSVC detection and environment setup is very problematic in general. MSFT changes their directory/registry structure constantly. Behind the scenes MSFT uses installation records or RPC/COM interfaces to lookup build tool details. There are some decent CMake modules that incorporate vswhere.exe, a tool that performs the same MSVC build tool identification as mentioned earlier.

I spent a few months and rolled my own optimized toolchain and tbh, it just introduced more potential problems. Between CMake's variable scoping, policy usage and subprojects that aggressively modify CMake build tool variables, the reality is less you provide for build tool configuration the better. A CMake build should succeed with only a few build tool variables, e.g. CC/CXX/Generator/Arch. With regard to Linux cross-compiling, folks seem to tend towards MSYS or Cygwin clang/gcc builds. Use the standard 32bit/64bit terminal shortcuts. From personal experience I avoid the compiler specific terminal shortcuts.

Also, I stumbled on this kit recently: https://winlibs.com/

The kits are fairly light weight, well maintained and overall pretty solid.

Finally, if you want to try out a toolchain, I'd recommend checking this one out: https://github.com/MarkSchofield/WindowsToolchain

The author of WindowsToolchain has done a good job exposing the MSVC Clang driver (clang-cl) AND the MSVC GNU frontend (clang/clang++).

Hope this helps and good luck.

[wareya]

I recently had to build Jackalope, which depends on TinyInst, which depends on xed, and I had to build it in a mingw (msys2/ucrt64) environment. mbuild insisted on trying to compile with MSVC, which, of course, didn't work, and I basically had to hit mbuild's python scripts with a stick to get it to use gcc to produce a .a file. (I did eventually succeed.) When poking around mbuild to comment out suspect if statements etc. to get it to build the way I needed it to, it looked like mbuild was doing the oopsie of assuming that the platform you're compiling on is linked to the toolchain you're compiling with, which is just not true, especially not on windows. There didn't seem to be a way for me to force it to do the right thing without modifying it. cmake would be less problematic in this regard.

[ddkwork]

I am trying to bind for go

https://github.com/ddkwork/xed/tree/master/kits%2Fxed-install-base-2024-11-27-win-x86-64%2Fexamples%2Fexamples

[mrexodia]

The integration with CMake's ExternalProject that I shared in 2020 is still working great in 2024! https://github.com/LLVMParty/packages/blob/604e454db80f76a953b7ac34dd93d4b7fe4efa50/xed.cmake