.cmake Adaptor (detection modules)?

[htfy96]

The main reason why I still stick to CMake is its builtin modules which provide compiler detection/package search/etc. It is unnecessary to reinvent the wheels, so I was wondering is it possible to provide (at least kind of) .cmake support and wrap these modules into xmake utilities?

[waruqi]

xmake also provide these features, so I don't intend to wrap them.

[ghost]

Compiler detection of xmake is perfect

"Package search" is working on the way. You could have a look branch repo

Thanks for caring :smiley:

[htfy96]

Thanks for reply. I'm just a little bit worried that there might be a lot more to do than you think. Take C++ as an example, we may need:

• feature test: whether this compiler supports variadic template, or is it claimed supported but actually buggy?
• compiler detection: we may be running clang frontend of MSVC, and many other corner cases
• compiler*platform-specific options: including sanitize/floating point accuracy/etc.

Supporting .cmake may reduce some kind of work, I guess.

[ghost]

I'd like to have a quick look into it this week

[waruqi]

@htfy96 You can use option to detect types, includes, funcs, links, cflags, ldflags and etc ..

for example:

option("test")
    add_define_if_ok("ENABLE_TEST")
    add_ctypes("wchar_t")
    add_cincludes("setjmp.h")
    add_cfuncs("sigsetjmp", "setjmp") 
--  add_cfuncs("sigsetjmp((void*)0, 0)")
--  add_cfuncs("sigsetjmp{int a = 0; sigsetjmp((void*)a, a);}")
    add_links("pthread")

target("demo")
    add_options("test")
    ...

It will detect wchat_t, "setjmp.h", libpthread.a and sigsetjmp.

The demo target will define -DENABLE_TEST and link -lpthread if all tests are passed

[waruqi]

And you can add add_cxflags("-fsanitize=address", "-ftrapv") directly.

If compiler does not support these flags, It will ignore it automatically.

[waruqi]

xmake will detect msvc on windows by default automatically. If you want to use other compiler, you can set it manually. for example:

$ xmake f --cc=clang --cxx=clang --ld=clang
$ xmake

And it will detect clang, gcc, .. on linux/macos.

[htfy96]

@waruqi

IIUC, according to your example xmake currently has no plan for builtin feature detection based on compiler version, instead, it provides basic utilities like add_cxxfuncs to allow custom detection. The main drawbacks of the absence of builtin has_variadic_template/support_cxx_11 include:

• We have to compile during configuration for many times if we have a lot of orthogonal features to detect
• Some feature detections are not as simple as add_cfuns(sigsetjmp((void*)0, 0) and requires heavy templates and domain knowledge

Actually I'm more willing to see something like enable_sanitize_address which adds proper flags for every compiler supporting this feature. Hard-coded/compiler-dependent flags are not encouraged from my perspective, and I want to completely eliminate these in my build scripts.

I'm referring to Clang/C2 frontend of MSVC. Take a look into CMake's patch set, and you'll find that checking _MSC_VER is not enough.

[waruqi]

@htfy96 Yes, xmake currently has no plan for builtin feature detection based on compiler version.

We have to compile during configuration for many times if we have a lot of orthogonal features to detect

This is only configured and detected once

Some feature detections are not as simple as add_cfuns(sigsetjmp((void*)0, 0) and requires heavy templates and domain knowledge

This is not a problem, the detection rules can be improved more flexibly.

Hard-coded/compiler-dependent flags

xmake takes the gcc flags as the standard and will map to other compiler flags smartly.

So these flags (-fsanitize=addres, -fomit-frame-pointer) are not hard-coded/compiler-dependent, you can see #71

About enable_sanitize_address, why not do the testing directly in the code?

#if (defined(__has_feature) && __has_feature(address_sanitizer)) || \
        defined(__SANITIZE_ADDRESS__)
...
#endif

[htfy96]

@waruqi Thanks for detailed reply. One more question, how can I switch flags based on whether a code snippet can compile without error? For example, I want to detect constexpr support:

option("test_cxx11_constexpr")
    add_defines_if_ok("CXX11_CONSTEXPR")
    add_cxxfuncs("has_cxx11_constexpr{ constexpr int f(int x) { return x ? x+f(x-1) : 0; } constexpr int x = f(5); static_assert(x == 15); ")

option("test_cxx14_constexpr")
    add_defines_if_ok("CXX14_CONSTEXPR")
    add_cxxfuncs("has_cxx14_constexpr{ constexpr int f(int x) { int sum=0; for (int i=0; i<=x; ++i) sum += i; return sum; } constexpr int x = f(5);  static_assert(x == 15);")

This doesn't seem to work. Older compilers do not support __has_feature

[ghost]

Actually I'm more willing to see something like enable_sanitize_address

Now for flags adding, xmake uses gcc flags mapping

Advantage:

• easier to remember

Disadvantage:

• Some flags are not mapped yet. See #71 I wanna to add a flag for coverage but -coverage is not mapped. And compilers are difficult so some flags cannot be implemented on some compiler

I agree with generic flags like what you said as a second choice

[ghost]

A good problem @htfy96

Seems that not supported yet, right? @waruqi

[htfy96]

And consider this case: user specified -fsanitize=address in xmake.lua, but actually the machine is running an older version of C2 on MSVC(e.g. Clang/C2 in MSVC14 Update 1) which has not sanitize support. How do you deal with such kind of issue?

The initiative of this issue is that compatibility/detection is extremely difficult. Manual checking could be very time-consuming and complicated. It is difficult to implement your own compiler database like CMake from scratch as well, since if you want to support N compilers, each of which has K versions, and you have P features to check, the database will soon balloons to N * K * P. Therefore, the best choice in this case is to adapt existing CMake databases.

[waruqi]

@htfy96 add_cxxfuncs and add_cfuncs only used to detect if the function interface exists. I will add add_cxxsnippets() to detect other feature support in the future, for example:

option("constexpr")
    add_csnippets("...")
    add_cxxsnippets("constexpr int f(int x) { int sum=0; for (int i=0; i<=x; ++i) sum += i; return sum; } constexpr int x = f(5);  static_assert(x == 15);")

[waruqi]

And consider this case: user specified -fsanitize=address in xmake.lua, but actually the machine is running an older version of C2 on MSVC(e.g. Clang/C2 in MSVC14 Update 1) which has not sanitize support. How do you deal with such kind of issue?

If has not sanitize support, add_cxflags("-fsanitize=addres") will be ignored automatically when building project

[waruqi]

The initiative of this issue is that compatibility/detection is extremely difficult. Manual checking could be very time-consuming and complicated. It is difficult to implement your own compiler database like CMake from scratch as well, since if you want to support N compilers, each of which has K versions, and you have P features to check, the database will soon balloons to N K P. Therefore, the best choice in this case is to adapt existing CMake databases.

Therefore, even using the CMake database does not guarantee all compilers, versions and archs. And this need instant update compiler database to detect new features and versions.

The most stable way is that try to compile it to detect it.

[htfy96]

@waruqi Validity check of snippets cannot be bulk-executed since an error in one snippet can easily affect others.

The -fsanitize example actually targets the underlying problem compiler detection: in order to detect whether this flag is supported, you need to compile with it during configuration instead of mapping it manually in scripts, as what you just mentioned.

Moreover, some features cannot be detected at compile-time without compiler mapping. For example: <regex> in g++-4.8 defined all functions with empty implementation.

[ghost]

@htfy96 add_cxxfuncs and add_cfuncs only used to detect if the function interface exists. I will add add_snippet() to detect other feature support in the future, for example:

:+1: @waruqi

add_snippet() is great. Just like my way to check if libreadline exists in Makefile in #80 . I'd like to try writing a first implement these days

The initiative of this issue is that compatibility/detection is extremely difficult. Manual checking could be very time-consuming and complicated. It is difficult to implement your own compiler database like CMake from scratch as well, since if you want to support N compilers, each of which has K versions, and you have P features to check, the database will soon balloons to N K P. Therefore, the best choice in this case is to adapt existing CMake databases.

You're right @htfy96

in order to detect whether this flag is supported, you need to compile with it during configuration instead of mapping it manually in scripts, as what you just mentioned.

xmake does you say, compile with it during configuration to check whether a flag is available @htfy96

Moreover, some features cannot be detected at compile-time. For example: in g++-4.8 defined all functions with empty implementation.

Great example :+1: @htfy96

Maybe best solution is

• detection first, database second
• flag mapping first, generic flags second

[waruqi]

@htfy96

Validity check of snippets cannot be bulk-executed since an error in one snippet can easily affect others.

You can define one option with a snippet to detect only one feature.

The -fsanitize example actually targets the underlying problem compiler detection: in order to detect whether this flag is supported, you need to compile with it during configuration instead of mapping it manually in scripts, as what you just mentioned.

You also can use option to detect flags. Although I don't support it now, I'll improve it.

option("fsanitize_address")
      add_cxflags("-fsanitize=address")  <-- I'll improve it here.
      add_ldflags("-fsanitize=address")

[waruqi]

@TitanSnow

Just like my way to check if libreadline exists in Makefile in #80 . I'd like to try writing a first implement these days

I've detected it in xmake.lua, just old core/makefile doesn't support. You can see core/src/xmake/xmake.lua

add_cfunc("API", "readline", nil, {"readline/readline.h"}, "readline")

[waruqi]

@htfy96

Moreover, some features cannot be detected at compile-time. For example: in g++-4.8 defined all functions with empty implementation.

But many vendors will modify the compiler (for some cross-toolchains), even the compiler with the same version does not guarantee that many implementations exist or not.

So there is the same problem with the database.

[ghost]

Yep @waruqi but this is just a example that not so exact and other code snippets could be used

[ghost]

So there is the same problem with the database.

Database is not no use at all! @waruqi

[waruqi]

@TitanSnow @htfy96 So I'm still more inclined to use compile detection. Perhaps xmake's detection is not flexible enough, but I will improve it.

[htfy96]

@waruqi It is OK to just provide minimal feature detection support, I guess. Users have to deal with it something like Boost.config, though in many cases we don't want to introduce Boost dependency. Quite looks like the way of premake.

Thanks for your effort. It would be piles of work to implement proper compiler detection.

[waruqi]

@htfy96 😄

[ghost]

add_snippet() is great. Just like my way to check if libreadline exists in Makefile in #80 . I'd like to try writing a first implement these days

Sorry, but for me there's still trouble to implement add_snippet()

[waruqi]

@TitanSnow I will implement it in last these days.

[ghost]

@htfy96 I have quickly looked through *.cmake in cmake/modules

I found that your idea is great :+1:

Many things are useful like findQt.cmake

In fact trying to link with Qt in xmake might be troublesome or I don't get the way? @waruqi

And FindGit.cmake might be useful for you @waruqi . I know you recently work on branch repo requires git

If .cmake adapter comes up in the future, I think things will be easier and better.

In this way, I agree with @htfy96

My option is on-fly detection is the first hand, off-fly data & script is the second hand

This idea should be minded @waruqi

Does you have a preliminary way to implement the adapter, @htfy96 ? Or could you do some work on this and finally give a PR?

[waruqi]

@TitanSnow

And FindGit.cmake might be useful for you @waruqi . I know you recently work on branch repo requires git

Git is a necessary component for xmake require action, not just looking for it. So it's not enough to use FindGit.cmake. xmake will install winenv with git when using xmake require to ensure git exist.

xmake also support tool detection and only for internal detection. In the future I will provide public interfaces to find tools like find_tool , find_package ..

If .cmake adapter comes up in the future, I think things will be easier and better.

Xmake has its own design pattern, not depend on .cmake or wrap it. But I will provide similar functionality in the future.

In fact trying to link with Qt in xmake might be troublesome or I don't get the way? @waruqi

This is really a good feature. I will improve option to support simular feature.

[waruqi]

@htfy96 I have implemented add_cxxsnippet to detect compiler feature. for example:

-- define option for detecting constexpr feature
option("constexpr")
    set_languages("cxx14")
    add_defines_if_ok("CONSTEXPR_ENABLE")
    add_cxxsnippet("constexpr", "constexpr int f(int x) { int sum=0; for (int i=0; i<=x; ++i) sum += i; return sum; } constexpr int x = f(5);  static_assert(x == 15);")

-- add target
target("test")

    -- set kind
    set_kind("binary")

    -- add files
    add_files("src/*.cpp") 

    -- need constexpr
    add_options("constexpr")

[waruqi]

I'll focus on improving option to provide more flexible detection in the future, but now I need to finish xmake require first. Please wait some time...

for example, provide some builtin options in xmake/options directory, like .cmake/Modules

option("find_qt")
      on_check(function (option)
             -- check qt ...
             -- add links, linkdirs .. to option 
      end)

option("xxx_interface")
      add_links("xxx")
      on_check(function (option)
             -- try compile codes
             -- try run and test codes (empty implementation?)
      end)

[ghost]

Maybe just translate some .cmake files manually is enough to finish the goal

I found cmake is licensed under BSD-3 so there's no problem

It doesn't have to make a adapter, hand-work is enough @htfy96

[waruqi]

@TitanSnow But I don't like cmake's grammar and design, and don't want to be overly reliant on and use them.

[ghost]

For grammar, I do not like cmake's either

But at least we could quickly look through them and come up with some ideas

[waruqi]

Yes, it provides some good features and ideas. I will refer to them when I begin to improve option.

[waruqi]

But I don't want to parse them for fast implementation, I need design the whole architecture first. Even if it will delay some time.

[ghost]

Yeah, it might be best solution

[waruqi]

@htfy96 @TitanSnow

I've started to implement some detection modules.

I'll add a new interface has_feature to make it easier to judge all compiler features in the future. for example:

target("test")
     set_kind("binary")
     if has_features("const_expr") then
          add_defines("-DCONST_EXPR")
          add_files("src/*.cpp")
     end

     if has_features("sanitize_address", "floating_point") then
         ...
     end

And I will improve option and target to support them.

target("test")
     set_kind("binary")
     before_build(function (target)

           -- import find_library
           import("lib.detect.find_library")

           -- find zlib and add links
           zlib = find_library("zlib")
           if zlib then
                target:add_links(zlib.links)
                target:add_linkdirs(zlib.linkdirs)
           end
     end

Or

option("zlib")
     on_check(function (option)

           -- import find_library
           import("lib.detect.find_library")

           -- find zlib and add links
           zlib = find_library("zlib")
           if zlib then
                option:add_links(zlib.links)
                option:add_linkdirs(zlib.linkdirs)
           end
     end

target("test")
     add_options("zlib")

[waruqi]

detect modules roadmap:

• [x] basic modules (lib.detect.*)
  • [x] find_file
  • [x] by os.files
  • [x] find from winreg
  • [x] find_path
  • [x] by os.filedirs
  • [x] find from winreg
  • [x] find_library #133
  • [x] by os.files and os.filedirs
  • [x] find from pkg-config
  • [x] find cross-platform libraries with other archs
  • [x] find from winreg
  • [x] find_program
  • [x] find from which
  • [x] find from winreg
  • [x] find_toolchains (cross, toolchains dir, ...)
  • [x] check_features for (compiler, linker ..)
• [x] detect tools (detect.tool.*)
  • [x] git
  • [x] curl
  • [x] wget
  • [x] unzip/tar/gzip/7z ..
  • [x] git
  • [x] curl
  • [x] wget
  • [x] compilers
  • [x] gcc
  • [x] clang
  • [x] msvc(cl.exe)
  • [x] dmd
  • [x] rustc
  • [x] go
  • [x] cgo
  • [x] linkers
  • [x] gcc
  • [x] clang
  • [x] msvc(lib/link)
  • [x] ...
• [x] detect sdk (detect.sdk.*)
  • [x] xcode
  • [x] android ndk
  • [x] vstudio
• [x] detect flags (lib.detect.has_flags)
  • [x] find flags from xxx --help
  • [x] try running it with flags
• [x] detect snippets
  • [x] lib.detect.check_cxsnippets for c/c++
• [x] detect includes
  • [x] lib.detect.has_cincludes
  • [x] lib.detect.has_cxxincludes
• [x] detect types
  • [x] lib.detect.has_ctypes
  • [x] lib.detect.has_cxxtypes
• [x] detect functions
  • [x] lib.detect.has_cfuncs
  • [x] lib.detect.has_cxxfuncs
• [x] detect features (lib.detect.has_features)
  • [x] core.tool.compiler
    • [x] compiler.features
    • [x] compiler.has_features
  • [x] compiler language features
    • [x] cpp_const_expr
    • [x] c_variadic_macros
    • [x] c_static_assert
    • [x] ...
• [x] lua api in xmake.lua
  • [x] option.on_check(), option.after_check(), option.before_check()
  • [x] target:add() in before_build()
  • [x] option:add() in on_check()
  • [x] option.add_csnippet, option.add_cxxsnippet

[ghost]

winreg roadmap:

• [ ] get
  • [x] path parse
  • [ ] type support
  • [ ] REG_BINARY
  • [x] REG_DWORD
  • [x] REG_EXPAND_SZ
  • [ ] REG_LINK
  • [ ] REG_MULTI_SZ
  • [x] REG_QWORD
  • [x] REG_SZ
  • [x] error info
• [ ] set

[waruqi]

@htfy96 @TitanSnow I have implemented these features, please see: new-features-v2.1.5