Identify tooling providers that we need to request input from

[ruoso]

For the first milestone of the Modules Roadmap, we want to be able to have a project that builds modules locally in the project and is able to import std.

The first step of this process will be to collect input from a wide range of tooling providers in order to identify requirements in the ecosystem, for that we need to identify who we need to reach out to in order to get the requirements.

Compilers:

• msvc
• edg
• clang
• gcc
• Circle
• Zig

Standard Libraries:

• libstdc++
• libc++
• Microsoft STL

Build Systems:

• CMake
• build2
• buck
• buck2
• bazel
• hmake
• meson
• xmake
• MSBuild

Packaging Systems:

• RedHat
• Canonical
• Debian
• vcpkg
• Conan
• tipi.build
• Anaconda
• Spack
• pkg-config
• pkgconf
• build2
• Mamba
• pixi
• FreeBSD Ports
• NetBSD pkgsrc
• OpenBSD ports
• MacPorts
• Gentoo Linux Portage
• Homebrew
• Fink

Static Analysis Tools:

• clang-based
• gcc -fanalyze
• Coverity
• Fortify
• Github Advanced Security (codeql)
• Sonar
• PVS Studio
• ReSharper C++ (for Microsoft Visual Studio)

Documentation Systems:

• doxygen
• hdoc
• standardese
• doxypress
• Sphinx

IDEs:

• CLion
• Visual Studio
• vscode
• vim
• emacs
• Eclipse
• Qt Creator
• Compiler Explorer
• Xcode

The list above will be edited as the discussion progresses, please comment with suggestions to change this list.

[mordante]

Maybe list Standard Libraries projects too. They are separate projects.

[daminetreg]

Hi @ruoso, we are working actively on this at tipi.build to support modules well and have done some breakthrough, I would definitely see us on the list of packaging systems.

[mathstuf]

Build systems: meson and xmake

Packaging: Anaconda, Spack

Static analysis: GCC's -fanalyze

IDEs: Eclipse, Qt Creator

[bretbrownjr]

Compiler Explorer would be an IDE on this list, I think.

Circle and zig would be compilers.

Possibly pkg-config and pkgconf would count as "vendors"... maybe as "other".

There are a lot more package managers of course: homebrew, Arch/AIR, Gentoo, PYPI, etc.

What are the plans to use this list? Possibly the list would be different depending on what it would be used for.

[mathstuf]

Another analyzer: PVS Studio

[ruoso]

What are the plans to use this list? Possibly the list would be different depending on what it would be used for.

My plan is to create an issue for each one of the items in the list and get in touch with the vendors/implementers for each one of them to get their input on what would be needed in the ecosystem for it to be possible for them to support a project that does import std.

[Klaim]

Note that build2 should also be listed in the "packaging system", it provides both build-system and packaging manager.

[Klaim]

About Anaconda: I'm part of the Mamba team which implements Conda package management but in C++ (Conda was originally in Python but recently adopted Mamba's library) so maybe that's relevant. The original author of Mamba also created pixi which also support Conda packages. Note that I am not sure if this ecosystem of libraries is relevant in the context of C++ modules, as these packages can hold not only C++-made stuffs.

[jwakely]

Packaging Systems:

It's not clear to me what this section means. Does it mean actual packaging systems, or vendors of packages built by such systems?

* RedHat

Red Hat isn't a packaging system. RPM (as used by Red Hat, SUSE, and Mandriva descendants) is a packaging format, DNF and Zypper are package management systems.

* Canonical

* Debian

Canonical isn't a packaging system. Deb (as used by Ubuntu, Debian and descendants) is a packaging format, APT is a package management system.

If the purpose of this section is to list vendors who might need to be contacted to discuss support for packaging CMIs, there are also source-based systems like BSD Ports, Arch, Gentoo, Nix/NixOS, ...

Documentation Systems:

* doxygen

Also https://hdoc.io/ and https://github.com/standardese/standardese and https://github.com/copperspice/doxypress/

Would a Debuggers section make sense? I'm not sure if debuggers need any particular support for modules.

[grafikrobot]

Would a Debuggers section make sense? I'm not sure if debuggers need any particular support for modules.

That does raise an interesting aspect. Packages can come with source. And debuggers do benefit from knowing where the source is. So, yes, I think adding debuggers to the list is useful. Irrespective of needing to know about modules or not.

[bretbrownjr]

In a brainstorming pass like this, all sorts of input are constructive. I'd rather people be more focused on connecting ideas than making sure they are naming a packaging system, a package manager, or a package repository.

Part of the reason I feel that way is that I expect the feedback process will be a bit fluid anyway. In my experience in engineer coordination exercises, the communication mechanisms can vary widely depending on the community and how it works. To be effective, one set of stakeholders might work best in email threads, another in text chat channels, another in issues on repos, and a third might require direct customer interaction with a relevant vendor. So it could look like discussing an organization that sells software sometimes and specific source repo in others.

That's to say I like the flexibility of the mechanism that @ruoso proposes. We can retitle, merge, split, or close issues as needed. And if attempts to engage particular organizations don't result in desired feedback, we'll have some record of what happened at least.

That being said, if people have a hard time making sense of our issues, that's good feedback, and something can probably be tweaked in that case. Specifics about what was confusing would help in that case.

[ChuanqiXu9]

Would a Debuggers section make sense? I'm not sure if debuggers need any particular support for modules.

As far as I know, as long as the generated dwarf information is correct, there won't be an issue in debuggers. Even if there are problems, it should be the job of the compiler to emit correct DWARF information. I feel the concept of modules is transparent to debuggers.

[ChuanqiXu9]

For the first milestone of the Modules Roadmap, we want to be able to have a project that builds modules locally in the project and is able to import std.

I feel it is already the case. e.g., Boris can built a relative complex project with libc++'s std module now: https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2.

The first milestone in my mind is that, a newbie can write and compile a hello world example by using std module without understanding what is modules and build systems. Just like we don't need to understand what is standard library and how is that get included and linked when we wrote our first C++ hello world example.

(There might not be gap. I just want to make things more clear)

[mordante]

For the first milestone of the Modules Roadmap, we want to be able to have a project that builds modules locally in the project and is able to import std.

I feel it is already the case. e.g., Boris can built a relative complex project with libc++'s std module now: https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2.

Boris points out the current solution is kind of a hack. Build2 ships files copied from libc++, so for every LLVM release build2 needs to be updated to the latest state of libc++. I really like to have a portable method for (Standard) libraries and implement that in libc++.

[ChuanqiXu9]

For the first milestone of the Modules Roadmap, we want to be able to have a project that builds modules locally in the project and is able to import std.

I feel it is already the case. e.g., Boris can built a relative complex project with libc++'s std module now: https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2.

Boris points out the current solution is kind of a hack. Build2 ships files copied from libc++, so for every LLVM release build2 needs to be updated to the latest state of libc++. I really like to have a portable method for (Standard) libraries and implement that in libc++.

Yeah, understood. I just want to say that currently we're already able to import std in codes. For example, I believe the above sample can be compiled with CMake as the example you give in the libc++'s documents.

It is almost a wording issue for me. I think currently it is already able to do that. And what we want to do here is to make it easier to get accessed and lower the bar to import std for users.

[ChuanqiXu9]

FWIW, I created an issue to support std module in compiler-explorer: https://github.com/compiler-explorer/compiler-explorer/issues/5404

[RedBeard0531]

Would a Debuggers section make sense? I'm not sure if debuggers need any particular support for modules.

As far as I know, as long as the generated dwarf information is correct, there won't be an issue in debuggers. Even if there are problems, it should be the job of the compiler to emit correct DWARF information. I feel the concept of modules is transparent to debuggers.

One issue that will come up is how to refer to module linkage entities inside the debugger. Modules introduce a new form of namespacing separate from C++ namespaces. It is now possible to have more than one ::a variable in a program without violating ODR. But the language doesn't provide any way to disambiguate them because there are no contexts in C++ sources where they can both be named. However, I think a debugger will often want to provide access to both of them.

[ruoso]

One issue that will come up is how to refer to module linkage entities inside the debugger.

Those do get a different mangling, so the debugger should be able to tell them apart. We do need to consider how we represent the module linkage in the demangled representation of the symbol, but that's not something that has needed convergence before, as it's just meant for human consumers.

[ruoso]

Thanks everyone for the suggestions and feedback. I think this is probably a good starting point for this project. I am now moving to #30 where I will formulate a questionnaire to serve as jumping point for the conversation with each of the tooling vendors.

[ChuanqiXu9]

I just got that if we forget ccache? ccache can't work with modules without refactoring. I filed an issue here: https://github.com/ccache/ccache/issues/1252

And also distributed build tools like distcc (there may be more).

[mathstuf]

sccache exists too and would need support.

[olgaark]

The Build section is missing MSBuild. It already supports STL modules with /std:c++latest though.

[RandomDSdevel]

For the first milestone of the Modules Roadmap, we want to be able to have a project that builds modules locally in the project and is able to import std.

The first step of this process will be to collect input from a wide range of tooling providers in order to identify requirements in the ecosystem, for that we need to identify who we need to reach out to in order to get the requirements.

⁝

     I'd also like to suggest some additional candidates for some of these lists, if I may. I'll take this one section at a time for each one I have potential candidates for. (I'll only focus on relatively prominent/popular tools.)

⁝ (Some sections snipped…)

Build Systems:

• CMake
• build2
• buck
• buck2
• bazel
• hmake
• meson
• xmake

⁝

• GNU make
• Non-GNU make implementations from other Unix distributions and distribution families:
  • BSD make:
    • FreeBSD make
    • OpenBSD make
    • NetBSD make
• GNU autotools
• Ninja
• Apple's llbuild, used by Xcode (for all languages it supports, not just Swift)
• Anything else that might be relevant from the Wikipedia article 'List of build automation software' (though this includes tools for more programming languages than just C++, of course.)

⁝

Packaging Systems:

• RedHat
• Canonical
• Debian
• vcpkg
• Conan
• tipi.build
• Anaconda
• Spack
• pkg-config
• pkgconf
• build2
• Mamba
• pixi

⁝

 (    (I won't touch on the naming debate that's started for this section; @bretbrownjr's response on that subject is fine for now.)

• FreeBSD Ports (and also derivatives, both actual and spiritual:
  • NetBSD pkgsrc (Also available for other platforms.)
  • OpenBSD ports
  • MacPorts
  • Gentoo Linux Portage
• Homebrew (Its builds mostly invoke shell commands, so support here should be easy.)
• Fink (for macOS)
• Any other relevant package distribution and OS platforms from:
  • The Wikipedia article 'List of package management systems'
  • The list of package ecosystems catalogued on Repology.

⁝ (Section snipped…)

Documentation Systems:

• doxygen
• hdoc
• standardese
• doxypress

⁝

• Sphinx (Can also work with Doxygen)

⁝

IDEs:

• CLion
• Visual Studio
• vscode
• vim
• emacs
• Eclipse
• Qt Creator
• Compiler Explorer

The list above will be edited as the discussion progresses, please comment with suggestions to change this list.

• JetBrains extensions for other IDEs:
  • ReSharper C++ (for Microsoft Visual Studio)
• Xcode

[mathstuf]

make, ninja

These are not build systems (well, make can be one, but I don't think it will survive C++ modules in any meaningful way), but build tools (or "executors"). ninja is explicitly not one and doesn't want to be one; I think we can leave these off of the list at least.

There are ninja implementations which may qualify; particularly shake.

[ruoso]

I think I agree with @mathstuf on the take on make and ninja. Make is a programming language and pretty much everyone writing a makefile is creating a build system of sorts using that language.

From the point of view of gathering input, I don't know if the maintainers of those would have much opinion on what C++ Modules require.

[jm4R]

KDevelop IDE