search

higan-emu / higan

higan is a multi-system emulator focused on accuracy, preservation, and configurability.
Other
1.18k stars 112 forks source link

Migrate to a more versatile build system such as Meson, CMake, or Autotools #8

Closed vilhelmgray closed 4 years ago

vilhelmgray commented 4 years ago

Right now higan relies on GNU Make to build its files and components. This has some shortcomings such as a rigid configuration, lack of dependency checking, and POSIX platform expectations just to name a few. GNU Make is not an inherently bad tool -- in fact it's quite the opposite -- it's just made to solve a specific problem in a specific environment, and it does that job very well.

Unfortunately, higan has a much larger deployment base and thus requires a more versatile solution. Modern build systems were developed to solve these problems, by codifying dynamic configurations for a diverse variety of platforms, optimizing parallel build paths and executions, and solving problems involving exotic environments. We should take advantage of the work done by these projects and utilize their tools to avoid duplicating the efforts of those before us.

To that end, I am recommending a migration to the Meson build system (although CMake, Autotools, or similar are good alternatives). I think Meson would be a good fit because of its philosophy of simplicity and speed. The code is especially terse -- just take a look at this example Meson file from the tutorial:

project('tutorial', 'c')
gtkdep = dependency('gtk+-3.0')
executable('demo', 'main.c', dependencies : gtkdep)

A number of prominent open source projects have also switched to Meson such as GNOME, Systemd, and Mesa. So I believe this is a robust build system for higan to use.

Github issue #1 and #2 can be addressed naturally by this migration as well.

orbea commented 4 years ago

If the build system it going to change I suggest against meson and would suggest sofort instead. That said I have a hard time seeing the current build system changing.

https://git.midipix.org/sofort/

Examples of sofort in use.

https://git.midipix.org/slibtool/ https://git.midipix.org/python/sbpython3/ (External build system for python3)

vilhelmgray commented 4 years ago

I'm not too attached to Meson so any modern build system would be acceptable for me. But I never heard of sofort, so out of curiosity why do you recommend it?

vilhelmgray commented 4 years ago

After investigating a bit further, perhaps CMake might be a better choice over Meson: https://www.rojtberg.net/1481/do-not-use-meson

CMake has a much larger community spread and native support for platforms such as Microsoft Visual Studios. A lot of dependencies already have built-in support in CMake so we wouldn't need to write our own checks for those either.

orbea commented 4 years ago

But I never heard of sofort, so out of curiosity why do you recommend it?

Its simple, portable, requires few external dependencies (posix shell, gmake) and strives for correct behavior. Autotools and cmake would drop the specific light requirement on gmake, but are much heavier and more complicated with many caveats.

That said the current build system is already familiar with developers and I'm not sure how much gain there would be from changing build systems now when such a change likely would of not happened with byuu?

vilhelmgray commented 4 years ago

That said the current build system is already familiar with developers and I'm not sure how much gain there would be from changing build systems now when such a change likely would of not happened with byuu?

I'm coming from a perspective as a Linux package maintainer and one of my concerns is that the existing method of manual gmake files will pigeon-hole packagers to require specific configurations for their systems in order to build and install properly.

For example, there are several hard-coded options in the higan GNUmakefile that have a syntax expecting a GCC compatible compiler and linker:

https://github.com/higan-emu/higan/blob/c19104ddd2f7ad1f8600a76866ca4df2d9d9270a/higan/GNUmakefile#L17-L24

Similarly, the install directives are hardcoded and make assumptions about the layout of a users' filesystem structure (granted with configurable top-levels):

https://github.com/higan-emu/higan/blob/c19104ddd2f7ad1f8600a76866ca4df2d9d9270a/higan/target-higan/GNUmakefile#L47-L55

These build and installation requirements do not need to be so rigid if a modern build system is utilized. This is the reason projects like Autotools were formed -- to consolidate knowledge about various systems so that projects could avoid hardcoding solutions and enable users to make use of flexible build environments.

You're right that current developers don't need a new build system when the current gmake files are tailored to their existing environments, but by restricting the build to a specific environment it inadvertently makes it more difficult for others to utilize those makefiles. If we have a more versatile build system, the code can reach a larger community of developers and users, with less friction along the way.

AndersonTorres commented 4 years ago

CMake has a much larger community spread and native support for platforms such as Microsoft Visual Studios.

In fact there is support for Visual Studio in Meson:

https://mesonbuild.com/Using-with-Visual-Studio.html

A lot of dependencies already have built-in support in CMake so we wouldn't need to write our own checks for those either.

Well, as far as I have studied, Meson supports many things including pkg-config. It isn't a "game changer" here.

And, well, Meson looks way cleaner to me than CMake.

AndersonTorres commented 4 years ago

I'm coming from a perspective as a Linux package maintainer

I am a package maintainer too, hehe! In fact I am a package maintaine of a very unorthodox Linux distro, NixOS, and its almost portable package management system, Nixpkgs.

Being very terse and fuzzy here: in order to support a plethora of configurations, NixOS installs every package in a dedicated directory inside /nix/store and configures symlinks in order to invoke them properly. It permits to maintain side by side things like "multiple version of GCC", or "two instances of Python compiled with different C compilers"... or, in the case of higan, "multiple higan instances with different GUIs".

In this sense, the current way higan is built is a bit unpleasant. I need to "manually" install data files and binaries in their locations instead of using a simple "make install".

If, by anything, I am interested in contribute for that.

vilhelmgray commented 4 years ago

@AndersonTorres Perhaps we can collaborate then if you like. I have a few projects I'm dealing with currently, but I have some free time in a couple weeks that I could use to attempt a migration to a modern build system -- and luckily the current Makefiles for higan are relatively straightforward so I don't anticipate much troubles ahead.

AndersonTorres commented 4 years ago

Well, I am trying to figure out how higan is built. There are some folders acting like libraries, and three executables.

Also, as far as I remember, some of those folders are duplicated in bsnes project. What about fusing them, or at least some factoring?

Also, there are a bunch of things that look like "libraries", as nall and hiro. It would be good if we treat them that way. It is very strange to enter in those directories and not being able to compile them. Also, some things are compiled multiple times.

Screwtapello commented 4 years ago

higan's current build system already builds on every supported platform (FreeBSD, Linux, macOS, Windows), so it's unclear what benefit an alternative build system would provide.

higan's current build system already provides many compile-time options. They're not documented very well, and they're often gratuitously different from platform conventions (for example, compiler instead of CC or CXX) and an alternative build tool might make them easier to find, but documenting the build options would be a lot easier than rewriting the build system.

For example, there are several hard-coded options in the higan GNUmakefile that have a syntax expecting a GCC compatible compiler and linker

There's also a lot of source code in higan that requires a GCC compatible compiler and linker. Luckily, the only portable compilers anyone cares about are GCC compatible (being GCC and Clang), so this isn't really a problem in practice.

Similarly, the install directives are hardcoded and make assumptions about the layout of a users' filesystem structure (granted with configurable top-levels)

The big problem with higan's current install directives is that it expects everything to be present in a single hierarchy, which must be writable. There's no way to split things into "read only" data in /usr/share and "read write" data in ~/.config or ~/.local.

I don't expect higan's existing install directives to be any help to third-party packagers, except as documentation on what they'll have to re-implement themselves. An alternative build system wouldn't really help here.

if there's a really good reason to switch build systems that nobody's mentioned yet, I'm interested to hear about it, but right now I'm leaning toward WONTFIX. I definitely agree that higan needs more documentation around the issues of compilation and installation, and that it should more closely follow platform conventions about managing configuration. I think those issues should be filed and tracked separately, though.

AndersonTorres commented 4 years ago

My two dollars:

1 - The higan build system does many assumptions that break in a more controlled system like NixOS. Supposing that alsaLib and udev are present just because platform==linux is not warranted. These things are optional at the very least, since FreeBSD has no such a thing as alsaLib.

2 - If the options need to be documented, maybe it also requires the rewriting of the build system, at least in some degree. If not in a full-fledged external software, at least by the current makefile-based approach.

vilhelmgray commented 4 years ago

Here's another quick example where assumptions are made about Linux that aren't necessary true for all systems: https://github.com/higan-emu/higan/blob/4e934754058c552abfea00494561be665e4998a4/nall/dl.hpp#L42

This hardcodes the lib directory at /usr/local/lib/ when it could be located alternatively in another popular path such as /usr/lib for example. With a modern build system, these kinds of common paths can be handled automatically during build configuration and provided to the higan files as a preprocessor DEFINE for the compilation.

In Gentoo, we have to manually sed these files in order to handle such issues before packaging higan for end users. I expect migration to a modern build system will significantly reduce this work by streamlining and handling a lot of the portability configurations.

AndersonTorres commented 4 years ago

This hardcodes the lib directory at /usr/local/lib/ when it could be located alternatively in another popular path such as /usr/lib for example.

In Nix it could be even worse. Nix is, at least in principle, platform-agnostic. It can be installed as an alternative package management system alongside another already installed operating system such as Ubuntu or Slackware.

All software installed by NixOS lives in a place called /nix/store, and we strive to guarrantee it doesn't call the original system libraries.

By hardcoding, higan breaks this barrier.

Screwtapello commented 4 years ago

Supposing that alsaLib and udev are present just because platform==linux is not warranted. These things are optional at the very least, since FreeBSD has no such a thing as alsaLib.

Luckily, on FreeBSD platform==bsd so it doesn't matter what the Linux platform defaults happen to be. And regardless, these are just defaults. If you build with ruby=video.xshm audio.openal input.sdl you'll get a binary that only includes those drivers, no matter what platform you happen to be building on.

This hardcodes the lib directory at /usr/local/lib/ when it could be located alternatively in another popular path such as /usr/lib for example.

This is an unconvincing example for a number of reasons:

So far as I can tell, the argument is "if higan had been designed to take advantage of a modern build system, it would be more comfortable for Linux packages to tinker with". And in that hypothetical alternate timeline, I'm sure that would be true. However, switching to a new build-system is not going to retroactively redesign higan to take advantage of it; there would still need to be a lot of changes to the source-code anyway.

I'm going to WONTFIX this issue and close it, because I don't think there's anything else we need to cover here. Instead, here are some paths forward: