A unified interface for interaction between IDEs, build systems, compilers and linkers

[KOLANICH]

Project description

There are many build systems for C++. All of them are crap. And all of them have problems with the way compilers and build systems interact.

• different compilers have different flags for the same functionality. This causes many-to-many problem when all build systems have to implement code to handle each compiler.
• doing the same work twice. There are many compilation commands, so many compiler processes are started, each redoes some work on startup. A better approach is to start a compiler once and keep it in memory untill build is finished.
• build systems have to probe compiler features instead of just getting info about their support

So it is prorosed to design a common interface between compilers, linkers and build systems.

1. everything uses CBOR as a serialization format. CBOR subset can be transformed into JSON, so we will use JSON in this issue to discuss about the schemas in the standard.
2. everything in/outputs CBOR from/to a pipe of calling process which number is passed as a standardized CLI arg.
3. every compohent of a toolchain supports some introspection.
4. for compilers we need at least

• sources and headers and their mapping to modules.
• prebuilt modules
• language version
• language features
• optimizations
• platform-dependent features
• mitigations
• linker params (subdocument) `

Relevant Technology

• clang
• other langs modules

Complexity and required time

Complexity

• [x] Intermediate - The user should have some prior knowledge of the technolog(y|ies) to the point where they know how to use it, but not necessarily all the nooks and crannies of the technology

Required time (ETA)

• [x] Much work - The project will take more than a couple of weeks and serious planning is required

Categories

• [x] Developer Tooling

[remram44]

All of them are crap.

Please don't be so negative. A lot of build systems have tried to address exactly that, like CMake. Consider pointing out specific improvements or explaining why contributing to those projects is not a good option, please.

Your approach seems to depend completely on the hope that every tool will adopt a single standard. I don't think this is reasonable. Having plugins that can translate from this standard protocol to their own interface is much more likely to work (at least in the interim) but then you're just making a build system like all others...

many compiler processes are started, each redoes some work on startup. A better approach is to start a compiler once

Lastly, do you have evidence on that? I would expect that the initial startup of the tool itself would be negligible compared by the loading and parsing of source code, headers, libraries... In addition, multiple processes allow for easier integration of multiple tools and parallelization, which is a must-have feature on modern computers and easily supported by current systems and tools.

---

To be honest, while I see room for improvement in current tooling, this reads much more like a rant (not even supported by data) than motivation for an open-source project 😅

[KOLANICH]

why contributing to those projects is not a good option, please.

Contribution to CMake is really not a good option: they have put it into own GitLab with reCaptcha. Also noone really wants to write CMake modules because CMake language is terrible. But CMake has some properties without which any other system is unneeded:

1. it doesn't require python or VM 2.supports multiple compilers

There are some quite nice build systems without python or JVM, like bake, but they don't support multiple compilers and to bring there compiler a abstraction layer and feture probing is an additional large effort.

But compilers know which features they have. It should be easy to make them emit a blob describing their capabilities. And it should be easier is to just parse the info in machine-readable way rather than implement feature probing for each build system.

Of course some of these code can be just rewritten into C++ and made a shared lib, but it is not yet done.

There is a downside for this approach: feature probing shows what really works for test programs.

Your approach seems to depend completely on the hope that every tool will adopt a single standard.

Yes, and compilers that don't support it can be supported by a dedicated tool which does transforms the blobs describing what to do into CLI arguments for each compiler and does feature probing.

then you're just making a build system like all others...

When the proposal is implemented the code doing the same work should be removed from build systems.

Lastly, do you have evidence on that? I would expect that the initial startup of the tool itself would be negligible compared by the loading and parsing of source code, headers, libraries...

Yes, it is.

In addition, multiple processes allow for easier integration of multiple tools and parallelization, which is a must-have feature on modern computers and easily supported by current systems and tools.

Multiple threads feels better because state can be shared between them easily and because spawning a thread makes less overhead than spawning a process. C++ modules proposal addresses some of these issues, BTW, though in CLang (according to the docs) the state is shared between multiple processes by using serialization.

[remram44]

Multiple threads feels better because state can be shared between them easily and because spawning a thread makes less overhead than spawning a process

The fact that it feels better doesn't do us much good, unless you actually have data that this makes a difference. Like I said, compiler startup is most probably a non-issue performance-wise. Nonetheless, if your "project" is to rewrite all the compilers and tools in the world to use a different architecture, I don't think it's reasonable at all.

I just can't tell why you opened an issue here. What do you expect this community to do? Why not open an issue with CLang and GCC directly? This is not a new open-source project, this is a wishlist for existing tools (and one we both know they won't buy into).

[KOLANICH]

In fact the main problem here is many-to-many and lack of introspection. Replacing processes with threads and reusing stored state is just a nice side effect. The problem here is that it may require some redong of what how compilers work, but I don't feel like it is very extensive. Additions require to just parse the blob, iterate over the jobs in it, start a thread for every job within a limit, when a job is processed, put a report into a qeueu. It is to get what ninja currently does. To get better some state sharing between threads is needed.