switch build system?

[stsp]

Currently (in master) fdpp is only buildable on focal and groovy. Bionic misses llvm-objcopy so that binutils is needed, and Xenial misses lld, so binutils is needed. There are a few ways to solve that:

1. Revert back to objcopy(binutils) and forget about Xenial and about llvm-only build.
2. Revert back to objcopy+ld.bfd (binutils) and forget about non-binutils builds forever.
3. Switch to xmake and do non-binutils builds when possible.
4. Create fdpp-legacy
5. Do nothing

Case 3 can only work if you can, as @andrewbird suggests, say "lld | binutils" in Build-Depends.

I would personally think I'd go for 1 unless @jschwartzenberg moves things to 3. :)

[waruqi]

Also, if they are already installed on the system, xmake will preferentially find and use them from the system via pkg-config, or you can configure add_requires("apt::zlib-dev") to integrate it directly from an apt installation if an apt source exists.

[stsp]

If the dependencies have been committed to the xmake-repo repository

No, can I instead put them into the source tree itself? Like I do with m4 macros for autoconf.

Also, if they are already installed on the system

What if they are not, but are available via dnf/apt? Although perhaps this is a bad case, since xmake would need root to install those...

[waruqi]

If the dependencies have been committed to the xmake-repo repository

No, can I instead put them into the source tree itself? Like I do with m4 macros for autoconf.

put 3rd library source code to self project? xmake supports it too. see

https://xmake.io/#/package/local_3rd_source_library

Also, if they are already installed on the system

What if they are not, but are available via dnf/apt? Although perhaps this is a bad case, since xmake would need root to install those...

Unless you explicitly configure add_requires("apt::xxx"), xmake will go ahead and call apt to install it, and it will request the user root. I wouldn't normally recommend this either.

With add_requires("zlib"), xmake will just look for the system libraries, and if it doesn't find them, it will download them from the xmake-repo repository, but it won't call apt.

[stsp]

put 3rd library source code to self project?

No, just the xmake build recipes, for the case when they are missing on a target system.

[waruqi]

? I don't understand your need. I haven't used m4 much.

[stsp]

If the dependencies have been committed to the xmake-repo repository

But what I want, is not to commit anything into xmake-repo. I want to put xmake recipes into my project, how to build various deps (for which the xmake recipes do not exist). And I'd like xmake, if not found in already installed packages, to d/l sources for these deps (using the URLs that I need to specify somewhere), and use the supplied recipes to build them.

[waruqi]

If the dependencies have been committed to the xmake-repo repository

But what I want, is not to commit anything into xmake-repo. I want to put xmake recipes into my project, how to build various deps (for which the xmake recipes do not exist). And I'd like xmake, if not found in already installed packages, to d/l sources for these deps (using the URLs that I need to specify somewhere), and use the supplied recipes to build them.

Then you just need to put the package configuration definitions from the xmake-repo into your project's xmake.lua .

You can put your package to your project or your local repository

see https://xmake.io/#/package/remote_package?id=using-self-built-private-package-repository

projectdir
    - myrepo
      - packages
        - z/zlib/xmake.lua

add_repositories("my-repo myrepo")

add_requires("zlib")

or does not use repositories

package("zlib")
    ...
    on_install(...)
package_end()

add_requires("zlib")

target("test")
    add_packages("zlib")

[waruqi]

Or use your private github repositories

see

https://github.com/NazaraEngine/NazaraEngine/blob/9f88d8a7e891e327ba4d2fefdcc77c40e4b42097/xmake.lua#L271

[stsp]

Yeah, private per-project repo should be enough! So to summarize:

projectdir
    - myrepo
      - packages
        - z/zlib/xmake.lua

By doing so, I'll be able to only put zlib/xmake.lua into my project, this xmake.lua will contain the URL for source download (I don't want to put zlib sources neither in my project nor in any remote repo), and xmake will use that optionally, given zlib is not installed. Correct? :)

[waruqi]

Yeah, private per-project repo should be enough! So to summarize:

projectdir
    - myrepo
      - packages
        - z/zlib/xmake.lua

By doing so, I'll be able to only put zlib/xmake.lua into my project, this xmake.lua will contain the URL for source download (I don't want to put zlib sources neither in my project nor in any remote repo), and xmake will use that optionally, given zlib is not installed. Correct? :)

right, you can only set remote url of zlib source.

[stsp]

OK, sounds like this can indeed boost the cross-platform portability. As currently the only thing we can do (with autoconf) is to write "this package is missing, please install it"

[stsp]

OK, I think I need some kind of the recursive layout to start the conversion. The projects are too large, and the builds are too scripted, to start converting from top-level. But I can easily start from converting some small sub-targets. Initially I can invoke xmake from make for those sub-targets. But I wonder how those sub-targets would share the global configure parameters set by xmake? Is there somewhere a guide on writing the xmake recipes incrementally for the large project?

[stsp]

sub-targets would share the global configure parameters set by xmake?

Besides configure parameters, the question is also about build dir. If every sub-target is built into its own build dir, things won't work. Somehow all sub-targets need to respect the build dir of the whole project.

[waruqi]

OK, I think I need some kind of the recursive layout to start the conversion. The projects are too large, and the builds are too scripted, to start converting from top-level. But I can easily start from converting some small sub-targets. Initially I can invoke xmake from make for those sub-targets. But I wonder how those sub-targets would share the global configure parameters set by xmake? Is there somewhere a guide on writing the xmake recipes incrementally for the large project?

share global xmake configure parameters to make?

If you want to share configuration in xmake.lua, you can use includes() api.

see https://xmake.io/#/guide/syntax_description?id=multi-level-configuration

[stsp]

If you want to share configuration in xmake.lua, you can use includes() api.

But how is to get that working with the otherwise make-based system? So for example I want to convert just one sub-target. According to your link, I need the top-level xmake.lua which will include the component's xmake.lua. But since the entire project is still built by make, I'll need to build the converted component from its sub-dir (rather than from a top dir), and make sure its build directory matches the make's build directory, rather than to follow the xmake's build dir layout. Usually this means: build this component in current directory, locating its sources via the command-line parameter. Is this possible?

Something like this: xmake --build-dir=. --src-dir=$(SRCDIR) in a makefile. Is this possible?

[waruqi]

If you want to share configuration in xmake.lua, you can use includes() api.

But how is to get that working with the otherwise make-based system? So for example I want to convert just one sub-target. According to your link, I need the top-level xmake.lua which will include the component's xmake.lua. But since the entire project is still built by make, I'll need to build the converted component from its sub-dir (rather than from a top dir), and make sure its build directory matches the make's build directory, rather than to follow the xmake's build dir layout. Usually this means: build this component in current directory, locating its sources via the command-line parameter. Is this possible?

Something like this: xmake --build-dir=. --src-dir=$(SRCDIR) in a makefile. Is this possible?

see https://github.com/xmake-io/xmake/issues/3342

$ cd workdir
$ xmake f -P $(SRCDIR) -o $(BUILDIR)
$ xmake

[stsp]

Oh, that looks pretty recent. :) Well, so if we have the integration with the existing build, and can convert targets incrementally, then I should convert some target as a test.

[stsp]

Of course the main problem seems to be the sparse docs... But thanks for sitting here, fulfilling that gap yourself. :)

[stsp]

$ xmake f -P $(SRCDIR) -o $(BUILDIR)

Doesn't seem to work as expected.

$ xmake f -o .
checking for platform ... linux
checking for architecture ... x86_64
$ xmake 
[ 20%]: compiling.yacc thunk_gen.y
[ 40%]: compiling.lex thunk_gen.l
[ 60%]: linking.release thunk_gen
[100%]: build ok, spent 0.263s

I wanted to get ./thunk_gen, but instead I am getting linux/x86_64/release/thunk_gen. Also I didn't want to set the output dir with a separate command, that does some "configuration". I just wanted to set the output dir with the cmdline switch, to easily integrate the xmake's component into a makefile-based project.

[waruqi]

$ xmake f -P (SRCDIR)−o(BUILDIR)

Doesn't seem to work as expected.

$ xmake f -o .
checking for platform ... linux
checking for architecture ... x86_64
$ xmake 
[ 20%]: compiling.yacc thunk_gen.y
[ 40%]: compiling.lex thunk_gen.l
[ 60%]: linking.release thunk_gen
[100%]: build ok, spent 0.263s

I wanted to get ./thunk_gen, but instead I am getting linux/x86_64/release/thunk_gen. Also I didn't want to set the output dir with a separate command, that does some "configuration". I just wanted to set the output dir with the cmdline switch, to easily integrate the xmake's component into a makefile-based project.

target("xxx")
    set_targetdir("$(buildir)")

[stsp]

With that I am getting build/thunk_gen. And if I do set_targetdir(".") then running xmake from source dir gives me ./thunk_gen. But running it from builddir this way: xmake -P /home/stas/src/fdpp/fdpp/parsers gives me /home/stas/src/fdpp/fdpp/parsers/thunk_gen rather than ./thunk_gen in builddir. Maybe I should set target dir via option, rather than writing it into the xmake.lua?

[waruqi]

With that I am getting build/thunk_gen. And if I do set_targetdir(".") then running xmake from source dir gives me ./thunk_gen. But running it from builddir this way: xmake -P /home/stas/src/fdpp/fdpp/parsers gives me /home/stas/src/fdpp/fdpp/parsers/thunk_gen rather than ./thunk_gen in builddir. Maybe I should set target dir via option, rather than writing it into the xmake.lua?

you can define custom option. https://xmake.io/#/manual/configuration_option?id=defining-options

option("outputdir", {description = "the output directory"})

target("test")
    set_targetdir("$(outputdir)")

xmake f --outputdir=xxxx

[stsp]

Ah, in this case also this works: xmake f --buildir=. before xmake. However,

1. Is the typo in "buildir" intentional?
2. Can I set it without an extra step, whatever this "f" is?

I just want xmake -P /home/stas/src/fdpp/fdpp/parsers --buildir=.

[waruqi]

Is the typo in "buildir" intentional?

history reason, I don't like double d.

Can I set it without an extra step, whatever this "f" is?

you need config it first. xmake f/config --xxx=, then run xmake build command.

or

xmake f -o . -P /home/stas/src/fdpp/fdpp/parsers; xmake 

[stsp]

Can you think of some config-build combo mode for better integration with existing build systems, where the config step was already done behind xmake's back? In such mode the user provides as much as he wants via cmdline, the rest gets defaulted, and the build process starts. What do you think?

[waruqi]

Can you think of some config-build combo mode for better integration with existing build systems, where the config step was already done behind xmake's back? In such mode the user provides as much as he wants via cmdline, the rest gets defaulted, and the build process starts. What do you think?

Is that what you're talking about?

trybuild mode:

https://xmake.io/#/features/trybuild?id=cross-compile-fast https://xmake.io/#/features/trybuild?id=examples-of-compiling-other-build-system

[stsp]

No, quite the opposite. I am converting make-based build system to xmake target-by-target. This have nothing to do with trybuild. Instead, I have all configuration params already in a variables. So I just need to pass them to xmake, bypassing its own config step. Like this: xmake -P /home/stas/src/fdpp/fdpp/parsers --buildir=. In this example I want to pass the buildir to it without doing 2 invocations.

[waruqi]

No, quite the opposite. I am converting make-based build system to xmake target-by-target. This have nothing to do with trybuild. Instead, I have all configuration params already in a variables. So I just need to pass them to xmake, bypassing its own config step. Like this: xmake -P /home/stas/src/fdpp/fdpp/parsers --buildir=. In this example I want to pass the buildir to it without doing 2 invocations.

you can define custom option. xmake f --foo=xxx --bar=xxx

and use has_config/get_config to handle them in xmake.lua

With that I am getting build/thunk_gen. And if I do set_targetdir(".") then running xmake from source dir gives me ./thunk_gen. But running it from builddir this way: xmake -P /home/stas/src/fdpp/fdpp/parsers gives me /home/stas/src/fdpp/fdpp/parsers/thunk_gen rather than ./thunk_gen in builddir. Maybe I should set target dir via option, rather than writing it into the xmake.lua?

you can define custom option. https://xmake.io/#/manual/configuration_option?id=defining-options

option("outputdir", {description = "the output directory"})

target("test")
    set_targetdir("$(outputdir)")

xmake f --outputdir=xxxx

[stsp]

Yes, but this still requires 2 invocations. I've got that to work, yes. But what I propose is to think up something to avoid 2 invocations, because the configuration step was already done by another build system.

[stsp]

So when xmake runs on behalf of some other build system, maybe you can think up the way to bypass its own config step and instead get all the needed params from the parent build system.

[waruqi]

Yes, but this still requires 2 invocations. I've got that to work, yes. But what I propose is to think up something to avoid 2 invocations, because the configuration step was already done by another build system.

You can't avoid the xmake config call, it's needed for all xmake tasks such as xmake build, xmake run, xmake install. It is not possible to pass arguments repeatedly for each of these tasks individually. Even configure/cmake/meson requires a separate configuration task.

[stsp]

Why not to have the combo mode config+build, that can do 2 things with 1 call?

[waruqi]

As I just said, there are too many combinations such as config+build, config+build+run, config+build+install , config + build +package and more . It is impossible to define a large number of repetitive arguments for each combination of commands, it is very unmaintainable.

[waruqi]

On unix/posix systems, it is very lightweight to perform two process calls, so why does it have to be one? With xmake f -o output; xmake build and xmake build -o ouput, there's not much difference between them, and only a few characters less. But the maintainability advantage of a standalone config task is obvious.

[stsp]

Firstly, I have no idea what is "f". :) Secondly, my build system currently does: cd builddir && $(MAKE) $@ for every sub-target. It then translates to: cd parsers && xmake f -o . -P $(abspath $(srcdir))/parsers && srcdir=$(abspath $(srcdir))/parsers xmake

which is, as you can see, quite clumsy and full of duplication. I probably can get rid of srcdir here. Currently I do

    local srcdir = os.getenv("srcdir")
    if srcdir == nil then
        srcdir = "."
    end
    add_includedirs(srcdir)

[stsp]

Yes, seems like I can use add_includedirs("$(projectdir)") to at least get rid of srcdir. Then it looks not so bloated. :)

[stsp]

How can I create the target for the generated source files? The closest I could find in your docs is asn1, but it still builds the binary by pre-defined rules. What I need is completely custom rules to produce various source files. Are there the examples of this? make has the special notion for such "intermediate" targets.

[waruqi]

How can I create the target for the generated source files? The closest I could find in your docs is asn1, but it still builds the binary by pre-defined rules. What I need is completely custom rules to produce various source files. Are there the examples of this? make has the special notion for such "intermediate" targets.

autogen?

https://github.com/xmake-io/xmake/blob/master/tests/projects/other/autogen_codedep/xmake.lua https://github.com/xmake-io/xmake/blob/master/tests/projects/other/autogen_code/xmake.lua

[stsp]

Not quite. Yes, these examples demonstrate the custom rule, but still they do set_kind("binary") and produce the binary at the end. I also do need binary at the end, obviously, but don't forget that I am integrating into an existing build system. So can I do set_kind("generated_cpp") or alike, while the binary is produced by the parent build system?

[stsp]

I mean, the concept of intermediate targets. Even make has that.

[waruqi]

Not quite. Yes, these examples demonstrate the custom rule, but still they do set_kind("binary") and produce the binary at the end. I also do need binary at the end, obviously, but don't forget that I am integrating into an existing build system. So can I do set_kind("generated_cpp") or alike, while the binary is produced by the parent build system?

other build system generate cpp and call xmake to build it?

you can set always_added to add these generated cpp files. add_files("$(buildir)/autogen.cpp", {always_added = true}) https://github.com/xmake-io/xmake/blob/master/tests/projects/other/autogen_code/xmake.lua

or in your custom rule, you can call compiler to compile these generated files.

[stsp]

other build system generate cpp and call xmake to build it?

Opposite: xmake generates them, parent build system build on them. For that I envision something like set_kind("generated_cpp").

[waruqi]

other build system generate cpp and call xmake to build it?

Opposite: xmake generates them, parent build system build on them. For that I envision something like .set_kind("generated_cpp")

You can generate it in many ways

phony target + custom script

target("test")
    set_kind("phony")
    on_config(function (target)
       -- generate cpp
    end)

or in before_build, ...

binary + on_build script

target("test")
    set_kind("binary")
    on_build(function (target)
       -- generate cpp
    end)

it will override the whole build stage. you can ignore binary kind.

target + custom rule

custom task

https://xmake.io/#/manual/plugin_task

you can define a custom task to generate cpp, then call it in makefile

task("generate_cpp")
  -- TODO
   on_run(function()
   end)

xmake generate_cpp

custom lua script

https://xmake.io/#/plugin/builtin_plugins?id=run-the-custom-lua-script

/home/generate_cpp.lua

function main()
    -- generate cpp
end

xmake lua /home/generate_cpp.lua

[stsp]

While there are 1024+ ways to generate anything, either with or without xmake, what intermediate targets should do, is at least to handle the updates properly. That is, when some target A depends on the intermediate target B, then any change of pre-requisites of B should re-build B and A. No changes of pre-requisites of B should avoid re-building of B unnecessarily. So if I do xmake generate_cpp, as you suggest above, I guess it would just call my generator func which will re-build everything unnecessarily.

So what of the proposed ways actually treat the generated results as a target, with proper deps tracking?

[waruqi]

You can add dependency files to detect if it needs to be regenerated. https://github.com/xmake-io/xmake/blob/3bdfa2178d729709c40669ec9f2e361636f01eb3/tests/projects/other/autogen_codedep/xmake.lua#L15

Yes, these examples demonstrate the custom rule, but still they do and produce the binary at the end.

you can remove batchcmds:compile to only generate cpp. https://github.com/xmake-io/xmake/blob/3bdfa2178d729709c40669ec9f2e361636f01eb3/tests/projects/other/autogen_codedep/xmake.lua#L12

as you suggest above, I guess it would just call my generator func which will re-build everything unnecessarily.xmake generate_cpp

you can also use depend.on_changed() in custom task/scripts to check dep/changed.

https://github.com/xmake-io/xmake/blob/3bdfa2178d729709c40669ec9f2e361636f01eb3/xmake/rules/verilator/verilator.lua#L218

[stsp]

You can add dependency files to detect if it needs to be regenerated.

Thanks.

you can remove batchcmds:compile to only generate cpp.

But what to do with set_kind() in that case?

you can also use depend.on_changed() in custom task/scripts to check dep/changed.

Can I get this more automated? Instead of me copy-pasting the low-level xmake code that I don't understand (even if its quite obvious how to modify it, as you suggest), would it be possible for me to only define the conversion rule, define the intermediate target that can apply that rule, and have xmake to handle the rest, like proper "xmake clean", on-demand updates, and so on? Can I avoid duplicating that code?

[waruqi]

But what to do with set_kind() in that case?

If you set binary kind, it performs the default link behaviour. If you don't want linking, you can override on_build/on_link or set it to object kind to skip link stage.

Can I get this more automated? Instead of me copy-pasting the low-level xmake code that I don't understand (even if its quite obvious how to modify it, as you suggest), would it be possible for me to only define the conversion rule, define the intermediate target that can apply that rule, and have xmake to handle the rest, like proper "xmake clean", on-demand updates, and so on? Can I avoid duplicating that code?

Usually you just need to add the dependency file

https://github.com/xmake-io/xmake/blob/3bdfa2178d729709c40669ec9f2e361636f01eb3/tests/projects/other/autogen_codedep/xmake.lua#L15

rule("autogen")
    set_extensions(".in")
    on_buildcmd_file(function (target, batchcmds, sourcefile, opt)
        local sourcefile_cx = path.join(target:autogendir(), "rules", "autogen", path.basename(sourcefile) .. ".cpp")
        batchcmds:show_progress(opt.progress, "${color.build.object}autogen %s", sourcefile)
        batchcmds:mkdir(path.directory(sourcefile_cx))
        batchcmds:vrunv("echo", {sourcefile, sourcefile_cx})
        batchcmds:add_depfiles(sourcefile)
    end)

target("autogen")
    set_kind("object")
    add_files("src/*.in")
    add_rules("autogen")

Can I avoid duplicating that code?

use rules or define custom module and import it in script. https://xmake.io/#/manual/builtin_modules?id=import

[stsp]

set_kind("object")

But why "object", if its a text file?

Usually you just need to add the dependency file

Is this enough to handle xmake clean correctly? From your example its not clear to me how xmake would find out which files were generated, to clean them up properly. Also I am not entirely sure how to add such generated files as a deps for another target.

[waruqi]

But why "object", if its a text file?

only object, phony, binary, headeronly kinds. I don't want to add more kinds.

Is this enough to handle xmake clean correctly? From your example its not clear to me how xmake would find out which files were generated, to clean them up properly. Also I am not entirely sure how to add such generated files as a deps for another target.

you can add your custom cleanfiles . add_cleanfiles() in description scope. target:add("cleanfiles", "xxx") in script scope.

or you can define custom on_clean scripts.

[stsp]

So how can I add such generated files as a deps for another target? In the parallel build process, unless xmake tracks them properly, they may not yet be generated when their dep is started to be built.