switch build system?

[stsp]

Currently (in master) fdpp is only buildable on focal and groovy. Bionic misses llvm-objcopy so that binutils is needed, and Xenial misses lld, so binutils is needed. There are a few ways to solve that:

1. Revert back to objcopy(binutils) and forget about Xenial and about llvm-only build.
2. Revert back to objcopy+ld.bfd (binutils) and forget about non-binutils builds forever.
3. Switch to xmake and do non-binutils builds when possible.
4. Create fdpp-legacy
5. Do nothing

Case 3 can only work if you can, as @andrewbird suggests, say "lld | binutils" in Build-Depends.

I would personally think I'd go for 1 unless @jschwartzenberg moves things to 3. :)

[andrewbird]

I can swap Travis to use focal for both fdpp and dosemu2, which is probably the right thing to do now anyway.

Option 4 seems to align quite nicely with dosemu-legacy. Option 3 is my least favourite, as I'd prefer that parallel builds worked sensibly with make, but of course I don't have to maintain the build system.

Locally I'm still using Bionic simply because the Lubuntu on Focal is so awful after its switch to the Qt tookit, that it was unusable! I'm probably going to need to switch desktops to Xubuntu which breaks my heart as Lubuntu on Bionic is just right for me.

[stsp]

I don't prefer 4 here because, unlike in dosemu2, where old builds reduce both security and functionality, here nothing is changed at all. So such separation here makes much less sense than there.

make was never a problem here (unlike in dosemu2), so I am not a big fan of 3 too, but some configure step may be helpful, as it is turning out now:

• bfd/lld/gold
• nasm/yasm
• llvm-objcopy/objcopy
• clang/gcc (not any soon)
• sed/gsed (see parse_decls.sh where I probe for gsed by hands)
• various flavours of install (worked to a common denominator already)

Build options:

• debug/non-debug/extra-debug
• use/unuse sanitizers

I'd say, the list is not as small as it was when I thought "oh, 10-lines makefile will suffice".

[stsp]

I reverted to objcopy to get at least Bionic back. But I still think its time to get configure, as too many build-time uncertainty have accumulated.

[minexew]

@stsp why do you write everything as a Haiku?

It's quite difficult to parse.

[stsp]

I think you see the automatic line-wrapping laying on top of the manual ones. I am trying to keep the lines short to avoid automatic wraps from ever happening, but sometimes they happen nevertheless. Resulting in something like this. :)

[jschwartzenberg]

I'm not sure that happened, I think @minexew is just generally curious about your automatic-wrapping-fobia ;)

[stsp]

I think its time to push this forward again, as it seems we'll need to switch from lld to GNU ld, which doesn't support the cross-compilation for free. I.e. lld supports all targets at once, whereas GNU ld needs to be installed in a special cross-build form. I looked into xmake docs, and it says is has a package management of its own, as well as supporting the native package managers. apt is listed, but rpm/dnf - not. @waruqi are you planning the dnf/rpm support? Also I wonder how exactly I can benefit from that package management. For the LP/COPR work-flows I think it would be very cool if xmake could just generate the rpkg and debuild specs automatically, with all the needed deps. Basically I think this is more about an integration with native builders, rather with package managers. I.e. it would be cool if xmake could, besides installing deps and building the package on its own, also generate the needed spec, on which debuild/rpkg can be used. Can it do something like this?

[waruqi]

but rpm/dnf - not. @waruqi are you planning the dnf/rpm support?

It may be considered in the future, and you can open a feature request first. We now support other package managers in addition to apt, such as conan/vcpkg/pacman/zypper/portage.

Basically I think this is more about an integration with native builders, rather with package managers. I.e. it would be cool if xmake could, besides installing deps and building the package on its own, also generate the needed spec, on which debuild/rpkg can be used. Can it do something like this?

like cmake/cpack? We are also planning to implement a similar feature to generate deb/nsis/rpm packages, but it may take some time, maybe a few more releases. Next release we will start with an initial implementation of support for nsis packages. Then we'll move on to support other packages like deb, rpm, etc.

BTW, xmake has been in the debian and archlinux repository.  https://packages.debian.org/sid/xmake

[stsp]

Generating packages is probably not what I need. I am looking for generating only the specs for lp/copr/obs work-flows. Is something like that planned?

[waruqi]

Generating packages is probably not what I need. I am looking for generating only the specs for lp/copr/obs work-flows. Is something like that planned?

I don't understand the format of these files. Maybe you can say more detail, or you can generate them yourself by way of a custom task.

[stsp]

rpkg: https://github.com/dosemu2/fdpp/blob/master/fdpp.spec.rpkg deb: https://github.com/dosemu2/fdpp/blob/master/debian/control obs: https://build.opensuse.org/package/view_file/home:stsp2/fdpp/fdpp.spec?expand=1

I of course can generate them by hands, but:

1. the build system has all the needed info for them
2. whenever the dep is changed, I need to update 4 places!

[stsp]

We are also planning to implement a similar feature to generate deb/nsis/rpm packages

Note that you will very likely need to generate these specs anyway, if you want to build the appropriate package. Because the build tools need it. Of course unless you somehow can build packages by xmake "directly", bypassing the host tools. Which I don't think is possible, or?

[stsp]

https://github.com/mesonbuild/meson/issues/49 meson seems to be doing that since 2015.

[waruqi]

We are also planning to implement a similar feature to generate deb/nsis/rpm packages

Note that you will very likely need to generate these specs anyway, if you want to build the appropriate package. Because the build tools need it. Of course unless you somehow can build packages by xmake "directly", bypassing the host tools. Which I don't think is possible, or?

Yes, I meant to generate these specs for packaging. xmake doesn't generate package binary directly. It generates all kinds of specs and then calls debuild, nsismake and other packaging tools to generate it. In addition, it does better automation, such as automatically installing required tools(debuild, nsismake ...) and then calling them.

Of course, we can also tell xmake to only generate specs and not to auto-package them by using the policy configuration.

mesonbuild/meson#49 meson seems to be doing that since 2015.

We also have a related issue:

https://github.com/xmake-io/xmake/issues/1433

The custom api feature in the recent dev release is actually in preparation for this packaging feature https://github.com/xmake-io/xmake/issues/4276

And we're already planning to implement the initial nsis spec generation in 2.8.6, as well as some initial design of the packaging and configuration APIs. If all goes well, and I have enough time, I will start trying to implement deb and rpm spec generation and packaging in 2.8.7. Of course, I haven't decided which one to implement first.

[stsp]

Thanks, this all sounds promising. The only problem is that something like 5 years ago I was getting the same answers, and there is afaics still no xmake in debian or fedora... :)

[stsp]

Also you likely know that there is a small difference between rpm and rpkg specs. Copr uses rpkg, obs uses rpm, but rpm spec can be generated from rpkg spec, so it might be better to generate rpkg, not rpm spec.

[waruqi]

Thanks, this all sounds promising. The only problem is that something like 5 years ago I was getting the same answers, and there is afaics still no xmake in debian or fedora... :)

We've been working on historical issues and we just recently resolved a 2 year old issue. https://github.com/xmake-io/xmake/issues/1452

It will take more time because I'm the only one maintaining it.

Also, xmake has recently been in debian. https://packages.debian.org/sid/xmake

By the time ubuntu 24.04 is released, we can get it.

And xmake has also been in archlinux.

There is also a fedora packager who is helping us package xmake into the fedora repositories, although I'm not sure when I'll be able to get in yet.

[waruqi]

Also you likely know that there is a small difference between rpm and rpkg specs. Copr uses rpkg, obs uses rpm, but rpm spec can be generated from rpkg spec, so it might be better to generate rpkg, not rpm spec.

We'll deal with those differences when we start working on this.

[stsp]

24.04 ok, any ETA for fedora?

[stsp]

Ah, you mentioned fedora already, sorry.

[stsp]

You don't have Mantic builds on LP. Why is so?

[waruqi]

You don't have Mantic builds on LP. Why is so?

xmake was just uploaded to the debian repositories last week, and ubuntu 23.10 has been frozen.

[stsp]

I mean this: https://launchpad.net/~xmake-io/+archive/ubuntu/xmake Can you add Mantic build?

[waruqi]

I mean this: https://launchpad.net/~xmake-io/+archive/ubuntu/xmake Can you add Mantic build?

I haven't added it to the ppa package yet, it will be included in the next xmake release.

[stsp]

I don't understand. I enable the new builds with 1 mouse click. What's the use waiting for the next release, to just enable the build on LP? I simply can't apt-get it right now. Why to wait?

[waruqi]

I don't understand. I enable the new builds with 1 mouse click. What's the use waiting for the next release, to just enable the build on LP? I simply can't apt-get it right now. Why to wait?

I don't know how to enable it on the web fastly, at the moment I can only upload it via ci when a new version is released. https://github.com/xmake-io/xmake/blob/042c51144c1a918d75493a94094e29aeafd924ff/.github/workflows/linux.yml#L50

[stsp]

Hmm, not sure. Today I went to LP UI and enabled the "Noble" build of fdpp by just checking in the checkbox. I always did so before, too. I can't give you any hints, as the only thing I know to do on LP, is to mouse-click through their convoluted UIs. I am currently uploading my nasm fork there, finding the click-through ways, and am enabling the needed builds w/o problems.

[waruqi]

Hmm, not sure. Today I went to LP UI and enabled the "Noble" build of fdpp by just checking in the checkbox. I always did so before, too. I can't give you any hints, as the only thing I know to do on LP, is to mouse-click through their convoluted UIs. I am currently uploading my nasm fork there, finding the click-through ways, and am enabling the needed builds w/o problems.

I didn't find the corresponding checkbox, but I've uploaded it by modifying the script to use a temporary ci to upload it. You can try it again.

[stsp]

[stsp]

Let me show you an UI, its here. :)

[waruqi]

Let me show you an UI, its here. :)

Thanks, but I still can't find it, I don't know where is the daliy-build page.

[stsp]

Its in "Code" section, not "Overview".

[waruqi]

Its in "Code" section, not "Overview".

nothing

[stsp]

I think you need to create a "series" (trunk series), create a source import, and then create a package recipe. Every time I am doing this for another project, I spending hours for finding the correct click-through path. Its always a quest. But at the end it usually works. :)

[waruqi]

Oh, sounds a bit complicated, I don't have much time to work on this. About mantic, I've uploaded it via script and it's pending, wait a bit more and you should be able to download it.

[stsp]

Yes, downloaded, thanks. Btw, are there any writings that compare xmake to others? Yes, I've seen your own, but for me its too vague. Not very concrete, not enough examples, and too terse. Maybe there is something else to read? My problem is that I am not quite familiar with pretty much any build system, other than make and autotools. So I can't even formalize my needs and say "this system has what I need, and this doesn't". Because I don't know what I need. :) Maybe reading a comparisons will help me to set up my mind straight.

[waruqi]

Yes, downloaded, thanks. Btw, are there any writings that compare xmake to others? Yes, I've seen your own, but for me its too vague. Not very concrete, not enough examples, and too terse. Maybe there is something else to read? My problem is that I am not quite familiar with pretty much any build system, other than make and autotools. So I can't even formalize my needs and say "this system has what I need, and this doesn't". Because I don't know what I need. :) Maybe reading a comparisons will help me to set up my mind straight.

Yes, most of the articles are mine. You can read some or docs.

• https://github.com/xmake-io/xmake/wiki/C-and-Cplusplus-build-system,-I-use-xmake
• https://xmake.io/#/getting_started

But there are also some articles and videos written by others:

• https://www.youtube.com/watch?v=ar3muou4BQM
• https://www.youtube.com/watch?v=ydgcsv-hssY&t=587s

[stsp]

When I run xmake inside dosemu2 folder, it keeps silence for about 5sec, and then says:

note: configure found, try building it or you can run `xmake f --trybuild=` to set buildsystem (pass -y or --confirm=y/n/d to skip confirm)?
please input: y (y/n)

5 seconds for nothing?? Come on!

[waruqi]

When I run xmake inside dosemu2 folder, it keeps silence for about 5sec, and then says:

note: configure found, try building it or you can run `xmake f --trybuild=` to set buildsystem (pass -y or --confirm=y/n/d to skip confirm)?
please input: y (y/n)

5 seconds for nothing?? Come on!

If your project does not have any xmake.lua files, xmake will do some of the following things

1. It tries to detect if the current project matches any existing packages in the xmake-repo repository, and if it does, it tries to build it using its package configuration. If it matches, it will use its package configuration to try to build it. As long as it matches exactly, the success rate of such a build is almost 100%, and xmake will automatically patch any known build errors.

see https://github.com/xmake-io/xmake/issues/2859

Detection is a bit time-consuming because it has to load all the package configuration scripts, but on my machine it only takes 2.8s!

ruki-2:dosemu2 ruki$ time xmake
note: configure.ac found, try building it or you can run `xmake f --trybuild=` to set buildsystem
 (pass -y or --confirm=y/n/d to skip confirm)?
please input: y (y/n)
^C

real    0m2.884s
user    0m0.655s
sys     0m0.346s

5. If it doesn't match the corresponding package script, then it will continue probing known build systems (autotools, cmake ...) ) to attempt a build. This has a certain failure rate, but it usually simplifies a lot for the user, and it also helps the user to simplify cmake/autoconf compilation for cross-compilation, android/iphone, etc.

see https://github.com/xmake-io/xmake/wiki/Xmake-v2.7.2-released%2C-build-third-party-libraries-more-intelligently#building-third-party-libraries-more-intelligently

If you have written an xmake.lua project configuration file for your project, you won't have this 5s problem and it will finish compiling very quickly.

[stsp]

But I think silently spending 3-5 sec is unacceptable, no matter what actually happens. Given that its a "first start", it can produce quite a bad first impression. I am sure you can remove that lag, keeping all the features in.

[stsp]

And what if I don't have internet connection? Will it spend even more time trying to contact its repo?

[waruqi]

And what if I don't have internet connection? Will it spend even more time trying to contact its repo?

will not

[waruqi]

But I think silently spending 3-5 sec is unacceptable, no matter what actually happens. Given that its a "first start", it can produce quite a bad first impression. I am sure you can remove that lag, keeping all the features in.

I have improved it. you can try

xmake update dev
source ~/.xmake/profile

[stsp]

Thanks, I'll wait for a PPA update. Can you enable Noble PPA build in a mean time?

[waruqi]

Thanks, I'll wait for a PPA update. Can you enable Noble PPA build in a mean time?

I'll enable noble when it's released. Also, you should be able to download xmake directly from noble, even without using ppa.

[stsp]

Also, you should be able to download xmake directly from noble, even without using ppa.

Is it already possible?

[waruqi]

Also, you should be able to download xmake directly from noble, even without using ppa.

Is it already possible?

I think so, it's already in the debian repositories.

[stsp]

https://launchpad.net/ubuntu/noble/amd64/xmake/2.8.3+ds-2 Seems so, congrats. :)

[stsp]

On some systems, like android, there are lots of missing deps. Also as now I am using the patched build of nasm, most of every system will get a missing dep. So I wonder if xmake can deal with that task: I can provide it with PPA and copr repo of the needed deps, so it can try to install them natively. But on the systems where this doesn't apply, it should build the deps from sources. I guess for that I'll need to set up some repo with xmake recipes for such deps. What automation level does xmake provide for such scenario? Can it automatically choice from the list of repos, which ones are natively applicable to install the certain deps, and which are needed as a fall-back to build the dep in-place?

[waruqi]

If the dependencies have been committed to the xmake-repo repository and it supports android, e.g., zlib, libpng, etc., then you can use it. Then we simply need to configure it.

add_requires("zlib")
add_requires("libpng 1.6.x")

target("test")
    add_files("src/*.c")
    add_packages("libpng", "zlib")

xmake will download their source tar package, then build and install it automatically.

we need only run

$ xmake f -p android
$ xmake

like this:

ruki-2:test ruki$ xmake f -p android --ndk=~/files/android-ndk-r20b/
checking for architecture ... armeabi-v7a
checking for Android SDK directory ... ~/Library/Android/sdk
checking for Build Tools Version of Android SDK ... 33.0.1
checking for NDK directory ... /Users/ruki/files/android-ndk-r20b
checking for SDK version of NDK ... 21
note: install or modify (m) these packages (pass -y to skip confirm)?
in xmake-repo:
  -> zlib v1.3 [from:libpng]
  -> libpng v1.6.40 
please input: y (y/n/m)

  => install zlib v1.3 .. ok         
  => download https://mirrors.ustc.edu.cn/debian/pool/main/libp/libpng1.6/libpng1.6_1.6.40.orig.tar.gz .. ok
  => install libpng v1.6.40 .. ok      

ruki-2:test ruki$ xmake
[ 25%]: cache compiling.release src/main.cpp
[ 50%]: linking.release test
[100%]: build ok, spent 0.172s