Support shell expansion in project.json files

[chamaeleon]

When compiling programs that depend on larger libraries, like gtk, there are quite often several required libraries that one need to link with. On Linux this set of libraries can often be found by using the pkg-config program. For example, Gtk3 library directories and library files can be found using pkg-config --libs-only-L gtk+-3.0 (directories) and pkg-config --libs-only-l gtk+-3.0 (libraries).

c3c supports passing arguments on the command line to provide extra library paths and libraries to link with. However, this does either require the user to either remember what to type, or provide a build structure on top of the project configuration file.

My suggestion is to allow for shell expansion of string values in the project.json file. Instead of

...
    "targets": {
        "gtk_hello": {
            // Executable or library.
            "type": "executable",
            // Additional libraries, sources
            // and overrides of global settings here.
            "linked-libraries-add": [
                "gtk-3", "gdk-3", "pangocairo-1.0", "pango-1.0",
                "harfbuzz", "atk-1.0", "cairo-gobject", "cairo",
                "gdk_pixbuf-2.0", "gio-2.0", "gobject-2.0", "glib-2.0" ],
            "linker-search-paths-add": []
        },
    },
...

we would have something along the lines of

...
    "targets": {
        "gtk_hello": {
            // Executable or library.
            "type": "executable",
            // Additional libraries, sources
            // and overrides of global settings here.
            "linked-libraries-add": [ "$(shell pkg-config --libs-only-l gtk+-3.0)" ],
            "linker-search-paths-add": [ "$(shell pkg-config --libs-only-L gkt+-3.0)" ]
        },
    },
...

As pkg-config lists libraries as a single string "-lfoo -lbar -lbaz" while project.json uses individual strings "foo", "bar", "baz" withtout the -l prefix, it probably implies some kind of post-processing is required (trimming the prefix, and/or splitting up the resutling string into multiple strings), or that yet another JSON key is added to the project.json file format (this does not seem desirable to me, having multiple keys that do just about the same thing in this case but by different means).

A benefit of using pkg-config is that it will expand to any new library dependencies that may be added over time (Gtk3 is presumably not a good example of this, since it is so old by now) and project.json does not need to be updated to reflect this.

Given string evaluation of this fashion, I could also envision that some people might desire a way of inserting environment variable content, perhaps using "${MY_ENVIRONMENT_VARIABLE}".

Edit to add side-note: Providing pkg-config output on the command line for c3c build, etc., means every program specified in the project.json file (if there are more than one), will get all those libraries linked in, whether they need them or not, I think.

[lerno]

Does it make sense to have pkg-config integrated in the build instead? The reason I ask is that arbitrary shell access like "$(shell pkg-config --libs-only-l gtk+-3.0)" suggests that it needs to be restricted under "arbitrary code execution access", whereas integrated access would not need one.

[chamaeleon]

It would work fine as far as I'm concerned, if it does not seem to pollute the "namespace" of keys too much. Assuming it being a key to an array of packages, under the hood it could then use --libs-only-L for the library directories, and --libs-only-l for the library names, and append all the resulting names to the respective command line arguments.

As a side-note, I'm not sure if you do anything much on Windows or not, or whether you use vcpkg or not. In any case, it has a similarly named program, pkgconf which offers the same options.

[lerno]

I am not using windows much no, and I need some advice on how to flesh this out. The feature itself is not particularly complicated, but doing it right so it's nice to use is important. Aside from just adding to the existing -l and -L flags, there is also the csource link arguments that need to have their settings, so there are four project and library settings all in all.

[chamaeleon]

For csources, I guess it might be required to use pkg-config --cflags or pkgconf --cflags (assuming Windows supports is in place for vcpkg) for additional include directories and compile flags required by the package.

{
...
    "pkg-config": [ "gtk+3.0", "zlib" ],
...
}

would add flags to the respective header include directories plus flags, linker directories, and libraries commands by issuing the required calls and collect the output. Depending on platform, the key being present would invoke the appropriate program.

[data-man]

I already proposed using libpkgconf once before.

[lerno]

@data-man Yes, that's what I remembered too.

[chamaeleon]

I already proposed using libpkgconf once before.

I was not aware of pkgconf existing as a library as well. I have no strong attachment to what means is used, but I do feel something that allows you to specify a package name rather than the individual pieces of data will be quality of life improvement. Not knowing about about libpkgconfig, or how it might possible most easily be integrated with c3c, I don't have much to say about that, as long as it works (besides being very keen that any compiler or tool I use is well-supported on Windows as well).

[lerno]

One should note that I assume the use of pkg-conf will break cross platform compilation.

This might not be obvious, but it's trivial to build cross platform for windows – and almost as easy for MacOS. Anything, like compiling C source code or similar, will break that. That has to be taken into account. This is why integrating something into C3 is not as interesting if it doesn't work cross platform.

[lerno]

For csources, I guess it might be required to use pkg-config --cflags or pkgconf --cflags (assuming Windows supports is in place for vcpkg) for additional include directories and compile flags required by the package.

We could also start with a more modest solution that only does pkg-config for the compilation? And presumably it should be both for libraries and for projects.

[chamaeleon]

I have personally not done a whole lot of cross compilation, but from what I can tell, pkg-config and pkgconf offer support for it, in the form of setting environment variables to indicate where the main directories are with respect to package definition files, target system libraries, etc. I'm not at all sure how that might interact with what c3c already does with respect to compiling and linking for a different system.

As for what kind of partial steps might be a good starting point, I'm not entirely sure what "only does pkg-config for the compilation" means. Providing C compiler flags and include directories for C sources, but not the linking phase with library directories and library files?

[lerno]

I have personally not done a whole lot of cross compilation, but from what I can tell, pkg-config and pkgconf offer support for it, in the form of setting environment variables to indicate where the main directories are with respect to package definition files, target system libraries, etc.

I've done some cross compilation with pkg-config. Not a great experience to be honest. More like hitting-things-with-a-hammer-until-it-works kind of experience (i.e. fiddle with environment settings, then put everything in a build script that sets the environment settings just right, essentially what pkg-config was supposed to fix in the first place)

I'm not at all sure how that might interact with what c3c already does with respect to compiling and linking for a different system.

Probably not great. One of the under-appreciated features is that I can just build the same code for Window or MacOS merely by picking a different --target, IF I stick to libraries that have precompiled binaries for Windows and MacOS.

Linux is a different can of worms entirely, because of the insanity involved trying to make binaries that work across distributions even: I have done some research, but there doesn't seem to be any great solution. You can compile with musl, but I haven't delved into that deeply. Basically every time I look at cross compilation for Linux I get sad, very sad.

As for what kind of partial steps might be a good starting point, I'm not entirely sure what "only does pkg-config for the compilation" means.

Sorry, that was unclear: I meant: only pkg-config to get dependencies for compilation of .c3 files (and not C compilation).

[chamaeleon]

Since pkg-config is more or less targeting linking, and C flags for compilation, I'm not sure if trying to tweak it to support c3 files is suitable. That would seem to be more of a job for a c3 specific tool or internal build support (if built into the c3c compiler).

I just want to stress that from my point of view, having the support for it available takes nothing away from the current experience, be it cross-platform compatibility or not, it only adds some convenience when building on the target platform (don't want to speak to ease or feasibility of overriding locations to provide different sets of target platform libraries).

The .pc files exists on Linux systems (I don't use Apple hardware or software, so I can't speak to the use or availability of it there), and on Windows, vcpkg does make them available when installing packages that way. I would very much like to be able to take advantage of the dependency information generated by the contributors of installed packages, rather than manage them by hand for each project I want to build.

If one is trying to build something that does not rely on .pc files, I don't see any reason to use pkg-config related functionality in the first place, and you go about your business using the other, already available, means specifying directories, etc.

If one has some libraries that have not been installed in such a way that pkg-config/pkgconf information is available (on Windows, perhaps a user is simply not using vcpkg), I don't see myself creating .pc files and stuffing them into a location somewhere. I'd more than likely simply specify the location and libraries explicitly in the project.json file. I would not be worse of than I am now, but I could be better off if I used vcpkg (on Windows for arguments sake) and its information was usable from project.json, rather than laboriously managing packages on my own.

I mention Windows a lot, but I do also spend a lot of time in Linux, and I would be of the same opinions regardless of which platform is discussed, just to make that clear.

[lerno]

Just to clarify here, would it be enough to just to shell out with pkg-config --libs-only-L <...> on windows when the directive is seen (and corresponding vcpkg option on windows) and if it fails it fails and reports that the pkg config didn't succeed?

[chamaeleon]

On Linux you'd use pkg-config, and on Windows you'd use pkgconf, and if the option --libs-only-L is used for either, the output from such a command would be a string of -Lwhateverdir, for if the option --libs-only-l is used a string of -lwhateverlib. vcpkg is "just" the means most people would probably use to get pgkconf installed in the first place on Windows (at least I'm not aware of an alternative to vcpkg that would do this).

If the command fails, either pkg-config or pkgconf did not know about the package because the dependency is not installed, or the tool is not installed or in the search path for programs.