Replace buildlib with a better system

[Eloston]

Inspired by this comment.

buildlib has a fundamental design flaw: it's a huge, monolithic Python library that requires users to write a Python script to customize and invoke the build. It works for those who want to build the browser for themselves, but it doesn't work well when packagers want use their own packaging system; e.g. apply patches and run individual building steps themselves, run only certain steps.

Idea: Break up buildlib into multiple command-line tools:

• Lower-level utilities: To be used by higher-level utilities, or users who want to use their own building system. Tool-specific options are passed in as arguments; shared options are stored as environmental variables(?)
  • Resource assmebler: For a target configuration, prepare one or more of the following resources into specified locations & format:
    • Patches and the patch order file
    • Domain substitution list
    • Source cleaning list
    • Meta-build configuration flags
  • Domain substitution
  • Source cleaner
  • GN builder
  • Source code downloader and unpacker (including required dependencies; e.g. custom syzygy code for Windows)
  • Build environment checker (for configurations that don't have it; e.g. Windows, macOS, static Linux builds
  • Any other miscellaneous tool that's shared between plaforms
• Higher-level utilities: For users that don't need to customize too much
  • Meta-build script generator: Generates necessary files and directories for building packages on respective platforms; e.g. a debian directory for Debian, a PKGBUILD file for Arch, etc.
    • Should have platform-specific logic for generating the packaging files
    • Maybe add options to specify additional meta-build configuration flags, patches, and ninja targets
    • Implementation detail: Use template packaging files/directories, much like how it's done with Debian in buildlib
    • Implementation detail: should be a Python library like buildlib. Also include __main__.py, instead of placing code in the top-level script invoked by the user
  • All-in-one builder: Automates everything from downloading to the final packaging
    • basically buildlib, but less customizable
    • Implementation detail: if necessary, make this into a Python library too

This will add two new top-level directories in place of the buildlib directory, and replace build.py with two new scripts corresponding to the higher-level utilities.

The resources directory structure can be left alone. However, some metadata representing the relationships between resource directories and names of platforms should be stored in the resources directory. It keeps a cleaner separation between the data and the tools (unlike buildlib which is a fusion of both)

If you have some ideas to improve this, I would like to hear them!

[ilikenwf]

I like Python too, but you may be able to do all this much more easily just using a combination of standard utils and cmake? It can call the Google/Chrome/Ungoogled/Inox speicifc scripts that are 100% required without overcomplicating the amount of code you have to (re) write...

This goes double since CMake support is native in VS2015, and I think 2013? If not there's the CMake .sln generator.

[Eloston]

I like Python too, but you may be able to do all this much more easily just using a combination of standard utils and cmake? This goes double since CMake support is native in VS2015, and I think 2013? If not there's the CMake .sln generator.

I'm not sure how this will work. There aren't really any standard utilities on Windows that are the same as any other platform. And I would rather not depend on something provided by Visual Studio; I can't trust it to keep consistent with what it provides.

But the bigger question is: How would CMake be used? Isn't CMake akin to the meta-build systems GYP and GN (in terms of its usage and purpose)? I don't need something like that. What I need is a meta-meta-build system.

buildlib is the way it is now because I wanted a more powerful version of the little shell scripts I used to use to build ungoogled-chromium with. But it ended becoming a more meta version of SCons (without even knowing what SCons was until recently), which also suffers from difficult packaging. I could add a ton of options, but that would dramatically increase the complexity. I believe CMake also suffers from the same problem.

So my idea is based off of two ideas:

• The UNIX philosophy, in that it's better to have separate smaller utilities that do their jobs well. This makes it easier for packagers to use the utilities in a more flexible manner (much like standard system utilities).
• Meta-build systems, in that the build configuration and the tools that read them are completely separate and build configuration files should not be Turing-complete. This can make the code less error-prone, as well as make it possible to check for build configuration semantic errors. This might seem a bit overkill if this build layer will stay simple, but I think it'll still help some in the long-run.

Though I'm still quite new to this particular topic, so I would still like to hear what you had in mind.

[nopjmp]

CMake is akin to systems like GYP and GN.

I think the current system is fine, it just needs to be refined into something simple and easy to use. The main issue is supporting 3 different platforms with their own quirks. All other build systems are code oriented. I've dealt with custom python build systems before on another project. This is fairly simple compared to that monster and I'm able to easily understand what is going on.

I think some sort of configuration file would make it easier with the defaults being what works on "generic" Linux. Windows can then override binary names and such and would get rid of different constants in the build system. Python takes care of some things like paths for you which makes this easier to abstract.

It would be best to avoid deviating away from what Google is using for Chromium otherwise we risk of falling behind on updates which means leaving users vulnerable to exploits and bugs.

I do understand your point on simplifying the build system, but Windows makes it a pain.

[Eloston]

The main issue is supporting 3 different platforms with their own quirks. All other build systems are code oriented.

The thing is, packagers don't want something that's simple and easy to use. They want something that is flexible and can integrate well with their shell-based packaging scripts; hence the reason why I propose to split it up into multiple utilities. If I leave buildilb the way it is now and add a lot of configuration options, it would be closer to satisfying the flexibility requirement, but packagers still don't like using one monolithic build system to do everything (e.g. Debian doesn't like SCONS). Plus it's a pain to maintain a lot of configuration options.

If it wasn't for the packaging requirement, I would probably leave buildlib the way it is now and just add a JSON-based configuration file that users could modify.

It would be best to avoid deviating away from what Google is using for Chromium otherwise we risk of falling behind on updates which means leaving users vulnerable to exploits and bugs.

I won't be deviating away from that. This is just a meta-meta-build system so users won't have to apply patches or run the various source processing scripts themselves. The rest of the build process stays to how Google does it.

I do understand your point on simplifying the build system, but Windows makes it a pain.

My proposal doesn't seem too bad to implement on Windows. Once I start working on it (in the develop branch), it will probably make more sense.

[lenormf]

Maybe bitbake? It would take a massive amount of amount to migrate though.

[Eloston]

@lenormf Bitbake is more sophisticated than what I need. My file formats will remain relatively simple.

[Eloston]

What has been committed now is an initial working version that needs revision to meet other objectives. Not all of the configuration options are there yet, and some scripts are buggy or untested. Only the initial Debian support is in the build files generator right now.

EDIT: Should point out that the buildlib files will be removed once everything's been ported over.