cmake build confusion

[abrownsword]

I am trying to update my project from 6.x.x to 7.1.0 and am having troubles with the cmake build system. My project uses cmake, and I include 6.x.x as a subdirectory. My cmake provides the various PostgreSQL_xxxxx variables (using pg_config because the cmake FindLibrary stuff is just broken) and sets a few of the options (shared libs off, tests off, docs off). Switching to 7.1.0 is just a mess and the docs aren't much of a help. It also appears that the cmake is attempting to "install" libpqxx somewhere (I don't want that), and that target properties need to be set in order to select static library instead of shared library? How do I set the target properties before I add_subdirectory?

[tt4g]

You can use the CMake variable BUILD_SHARED_LIBS to make the library to be shared or static. e.g. set(BUILD_SHARED_LIBS OFF) to build static library.

The installation of CMake takes place when you run the CMake INSTALL target. If you run the CMake build step with the cmake --build . --target command, the installation will not be executed. Or you can use Add_ExternalProject() instead of add_subdirecotry() to avoid installing pqxx: ExternalProject — CMake 3.17.2 Documentation

Also, the installation location of the library by CMake is specified by CMAKE_INSTALL_PREFIX: CMAKE_INSTALL_PREFIX — CMake 3.17.2 Documentation

[abrownsword]

Add_ExternalProject is way more convoluted to use than add_subdirectory -- I'd rather not go there if I can avoid it.

CMake is complaining that PostgreSQL_TYPE_INCLUDE_DIR and PostgreSQL_INCLUDE_DIR are not defined. I have PostgreSQL_INCLUDE_DIRS (which libpqxx also seems to require?), and setting those other two to that one seems to make it work... why are there 3 needed/used?

By setting PostgreSQL_FOUND to true and BUILD_SHARED_LIBS to false, it seems to be happy now. Except for the API changes which I need to change my code to handle.

[tt4g]

CMake is complaining that PostgreSQL_TYPE_INCLUDE_DIR and PostgreSQL_INCLUDE_DIR are not defined. I have PostgreSQL_INCLUDE_DIRS (which libpqxx also seems to require?), and setting those other two to that one seems to make it work... why are there 3 needed/used?

This is specification of find_package(). When find_package(<PackageName>) is called, CMake looks for Find<PackageName>.cmake, <lower-case-package-name>-config.cmake or <PackageName>Config.cmake from CMAKE_PATH, etc. Usually <lower-case-package-name-config.cmake and <PackageName>Config.cmake are provided by the libraries installed by CMake. A third party provides Find<PackageName>.cmake for libraries that are not installed by CMake. Please refer to the official CMake documentation for more information: find_package — CMake 3.17.2 Documentation

FindPostgreSQL.cmake is officially provided by CMake. PostgreSQL_TYPE_INCLUDE_DIRand PostgreSQL_INCLUDE_DIRare variables defined byfind_path()` in this module. These variables are CMake cached, so you can override them. See source: https://github.com/Kitware/CMake/blob/v3.17.2/Modules/FindPostgreSQL.cmake

FindPostgreSQL.cmake module looks for released PostgreSQL at the time of CMake's release. If you are using an old CMake or have PostgreSQL installed in a location that CMake does not know about, CMake will not be able to find the library.

To work around this problem, in CMake version 3.12 and later, you can specify PostgreSQL_ROOT and CMake will use that path as the root of the library search path. Prior to CMake version 3.12, the workaround was to specify PostgreSQL_TYPE_INCLUDE_DIR and PostgreSQL_INCLUDE_DIR.

PostgreSQL_INCLUDE_DIRS is the set of PostgreSQL_TYPE_INCLUDE_DIR and PostgreSQL_INCLUDE_DIR.

[jtv]

I don't think I can do much to improve the upgrade experience from 6.x for CMake users at this point. But I'd like to put some more time into the documentation, and I'm open to any help you can offer for improving the CMake setup.

For starters, there's a lot of duplication between README.md and INSTALL. I'd like to fold those together. Maybe rename INSTALL to INSTALL.md, and have the build instructions in README just say "See INSTALL.md for instructions."

Also, I don't want to be explaining too much stuff that happens in third-party software such as CMake and autoconf. One, qualified users will already know it and two, that software can change and leave my documentation out of date. Where to draw that line is an eternal conundrum.

[abrownsword]

We definitely don't want or need documentation about the tools (i.e. CMake), but it is important to document clearly how this project uses them. For example, the externally visible variables (e.g. BUILD_SHARED_LIBS). A small example or two can often be more effective than descriptive prose. A snippet from a cmake project that uses add_subdirectory would demonstrate that this use case is possible and a working point to start from.

[jtv]

Absolutely agree. Next question: what would that snippet look like? CMake is not my strong suit, so I'm juggling contributions from more informed people here.

[abrownsword]

Well I'm not exactly an expert in cmake either, plus my project has some unusual aspects (which is why FindLibrary isn't working for me), but here's what I have that is working:

set(libpqxxdir "libpqxx-${LIBVERSION}")     # LIBVERSION set above
set(SKIP_PQXX_SHARED on)                    # obsolete now?
set(SKIP_BUILD_TEST on)
set(BUILD_SHARED_LIBS OFF)

# used instead of FindLibrary, PostgresSQL_INCLUDE_DIRS set externally
set(PostgreSQL_FOUND true)
set(PostgresSQL_INCLUDE_DIR ${PostgresSQL_INCLUDE_DIRS})
set(PostgresSQL_TYPE_INCLUDE_DIR ${PostgresSQL_INCLUDE_DIRS})

add_subdirectory(${libpqxxdir})

[jtv]

I'll see if @KayEss has some time to cast his insightful eye over this as well, so we can come to some good documentation. :-)

Still dealing with a bit of a personal crisis here, so I may be a few days.

[abrownsword]

Hope it resolves as quickly and cleanly as possible.

[KayEss]

There's some obvious points I think:

• Document the configuration that is added, like SKIP_BUILD_TEST and BUILD_DOC. It's probably worth making sure that these have PQXX in their names too, just to make it super clear.
• Mention the standard cmake configuration that is honoured:
  • like use of BUILD_SHARED_LIBS
  • That tests are registered with cmake
• Some notes on use of the project:
  • that installing results in being able to use FindPackage
  • it can be used with add_subdirectory
  • I expect it's worth mentioning that you may need the postgresql-server-dev-all package as well as libpq-dev (as that seems surprising to most of us). I'm not sure how this reifies on other platforms.
• Some platform notes...
  • On Unix this is pretty simple, but mac users probably need to be aware that brew installs libpqxx as part of the Postgres installation so the library order used is important.
  • On Windows users need to decide where they want include files and library files as there's no default (this is really the only difference -- every other aspect is remarkably similar). As such it might be easier for them to prefer the add_subdirectory way of using it and just let their IDE deal with it (if they're not using an IDE then most likely they have this well figured out).

That's all stuff off the top of my head.

[jtv]

Okay, I've moved the build documentation into two separate files: BUILDING-configure.md and BUILDING-cmake.md.

Now it's a matter of correcting and extending the latter.

[jtv]

@KayEss I have documented SKIP_BUILD_TEST, BUILD_DOC, BUILD_SHARED_LIBS as you suggested, as well as the need for the full postgres development package.

I'm still not experienced enough with CMake to be confident that I can explain the other points well. I don't yet know what it means for tests to be registered with CMake, what the implications of FindPackage and add_subdirectory are, or what you mean to say about library order for macOS.

[jtv]

@abrownsword I've incorporated your config snippet in a new section (called "Use") in BUILDING-cmake.md. Perhaps you could take a look and suggest fixes or additions?

[abrownsword]

I would add a simple 1-line example (i.e. just the add_subdirectory line) that shows normal usage which relies on FindLibrary and the default options. Then have my example which asserts more control. No need to mention me by name. :)

[GordonLElliott]

Several notes inspired by various things someone said here, and bit of my experience (strictly on Windows with Visual Studio):

One of the oddities is that the configuration step search for PostgreSQL (built into CMake how it does that) seems to need a complete PostgreSQL installation, not just the libraries. But what is needed is more or less just the libraries (and,... well we'll get to that), not a complete working PostgreSQL database system. Obviously the libpq library is needed from Postgres, and a working PostgreSQL system on which testing and applications can refer to need to be accessible. But there is no real reason that an entire installation of PostgreSQL should be needed.

Now on Windows, the standard PostgreSQL library also requires several "DLL" components, and the correct copies of those DLL components come from the executable 'bin' directory, not the 'lib' directory which has similar but not necessarily compatible versions of many of those same DLLs as well. (Explanation???) The PostgreSQL Windows' installation procedures have options for just the libraries--but that does not install the correct DLL files for WIndows operation. A complete installation more or less sets up a complete server on the user's computer which will start in the background--even if the user plans to uitilize an external server for the testing and applications.

Now as of late, doing clean installs using an "out-of-source build" the CMake has always chosen the correct PostgreSQL installation on my computer--but in earlier trials (in which I didn't understand CMake at all) I had CMake finding 32 bit Postgres to go with a 64 bit install and mixups like that. (And there may be explanations for that such that most users don't need to be concerned).

After working with the guys here we got necessary fixes so after version 7.0.7 one can use CMake flag -DPostgreSQL_ROOT=<location of Postgres> as argument for the first configuration step. However I have only tested that with a complete working PostgreSQL installations--as noted in prior paragraphs, and might not work with just the libraries installed as that location is apparently a hint to the find package operation. (It does work for both 32 and 64 bit installations on the same machine, by the way, so as to be able to correctly build and test each successfully on Windows.)

Using -DPostgreSQL_LIBRARY="<location>\lib\libpq.lib" -DPostgreSQL_INCLUDE_DIR="<location>\include" -DPostgreSQL_TYPE_INCLUDE_DIR="<location>\include" also works for me, and that should stop the automated search for PostgreSQL.

KayEss (Hi--we haven't "met") suggests that there is no "standard" installation location on Windows. There sort of is such a standard, not rigorously followed, but which is the standard where the PostgreSQL install uses. Assuming programs on drive "C", for libpqxx might be C:\Program Files\libpqxx\<version>\... (wherein the lib, include, and share subdirectories would occur--and sorry to offend with the Windows backslashes.)

However putting those files in that location has many problems. CMake (by itself), nor Visual Studio build process (by itself) will have permissions to do that final installation in the Program Files directories. Furthermore on a 64 bit operating system CMake makes the wrong decision of the naming convention and tries to put the 64 bit library into the C:\Program Files (x86) directory by default, which on a 64 bit operating system is reserved (mostly) for 32 bit programs. (Of course strangely enough the Visual Studio 64 bit compilers are found there ... but that is another of many horror stories). The CMake configures the Visual Studio build to put it in that (wrong) directory by default, then the install step (which is needed one way or another) will attempt to put it in that privileged location and will fail. Even logging in as admin to run does not confer the privileges. What is needed is a special configuration to run a batch file, CMake, or Visual Studio build system as administrator which is not simple like other operating systems.

Now as to "INSTALLATION":

The CMake build process has three parts (which may be combined in some instances to two steps), which are configure, build, and 'install'. However one must not misunderstand the importance and meaning of the 'install' step. That is the step in which a clean set of libraries, include files, and (optionally) documentation files are assembled. Those files are present somewhere in the build files, but that final 'install' step copies out the clean (more or less) set into organized and sensible (relative) locations. Even if you are not expecting the CMake build to "install" the files in their final installation location, the "install" process should probably be used to a local "install" subdirectory to produce the clean file set. (And yes, I'm sure that CMake can be used from a higher level to break out those steps, which I suspect is abrownsword's intent. So rather than ignore, I suspect abrownsword could hack the CMake "install" steps to configure them as desired.)

(Just bit of news: I'm working to complete my batch file "easy" install for Windows/Visual studio this weekend, and set up the repository. (Monday is a holiday in the United States, and we are avoiding other people so not a normal holiday.) And I'm quite certain this can all be converted to CMake like abrownsword is using, but would have to include doing several things that are specific to Windows and which would circumvent the normal CMake ways in order to fix some of the problems mentioned above. My setup will build libpqxx from the source, create a local "install" (2 copies one for Debug, one for Release if specified), set up final install with lib subdirectory broken into Debug and Release sub-subdirectories but only one set of include files, then make "property pages" to enable usage of the libraries in a project by just clicking that into the VS system for the respective build configuration of the user's project. That will include copying of the messy Postgres DLL files into the execution directory of the project, so one gets a working application. And it asks for permission to become admin so as to copy the final installation to the Program Files directory.)

[abrownsword]

Oh, I am so glad I don't need to work in Windows anymore.

Just to respond to my specific mention: I very strenuously dislike and avoid builds which need to be "installed". My particular project is an application that uses the libpqxx library, and the application needs to be as easily relocated and self-contained as possible while minimizing installed dependencies on the system. Hence my choice to use static libraries rather than shared libraries wherever possible. This is an important usage model, but clearly not the only one. This library's documentation should provide users with enough information about its build system to use it in the particular manner they require. Showing all possible use cases is impractical (if not impossible), but what is provided ought to show all available knobs that users can tweak to get what they want.

[GordonLElliott]

My application is a group of developers (well, at least 2) and several applications. So I'm making an easy way to set that up, so each new application can be converted to new techniques and thrown into a clean project. (I use the "property pages" method in which all the ugly work is hidden into a once-occurring configuration everyone uses.) Of course in my instance we do have some sort of "install" location (whatever one calls it) for the libraries and other files needed by the application. Otherwise sounds like we both have the same problem (I have more people and applications all to a common group, you have sounds like a single application that user's work with the source code). I don't know if you are making any of your CMake work open. I haven't figured out how on windows to make the PostgreSQL do static libraries without also controlling the PostgreSQL build itself, and I guarantee we have been through "DLL Hell."

I agree on documentation -- but be easy with these guys as I have gotten enormous amount of help from them even though they were very busy. I started without knowing the first thing about CMake, even thought it came with my Visual Studio. (It literally did--but an old version that was incompatible with the very version of Visual Studio in which it was supplied.) I'm really saying it may be partially up to us to find and help supply that documentation. I've been really lucky with my project, got fixes I needed in a few days.

OH, and further on that "install" point: I use the CMake to make a local directory that uses the CMake step called "install". But the meaning is simply the extraction of the needed output into an organized place. I put it in the relative directory tree along side of my source etc. In itself has nothing to do with "installing" the package in a specific location, and all to do with organizing the final output of the configure-build-[well.... 'output'] process. Yes I understand that many people use that as an actual "install" process--but on windows it is completely broken so one can't even use it that way directly from CMake--I have to make a local copy of the CMake process output for the sequence to not fail.

[jtv]

Thanks @GordonLElliott — still having trouble keeping up with what you're writing, but I appreciate any concrete improvements you can suggest. As you say, it may be partly up to you to find and help supply that documentation, because I'm a total amateur at CMake. :-)

@abrownsword also makes two interesting points there:

1. Microsoft's platform may be why I had a headache today.
2. With CMake you don't necessarily want to install the project.

I think I can just about figure out how to incorporate that last note in my updated documentation... I'll get things wrong though, and ask for feedback. :-)

[GordonLElliott]

Jtv, I completely agree. (And I already have a headache today... But should spend more time on doing instead of talking about what I will be doing...)

I want to emphasize again (for everyone) that the CMake "Install" step most importantly organizes the include files. Before the "install" step the include files do exist -- but with something like twice as many auxiliary files also. The "install" step (get to that in a moment) puts together an include subdirectory with only the files one needs. (I don't know how it chooses that, BTW, but it does!)

Now the "install" step can be gotten at by several ways. In the end it is really a CMake execution, for example:

cmake -S libpqxx-<version> -B binary
cmake --build binary --config Debug
cmake --install binary --config Debug --prefix "output/libpqxx/Debug"

With some error checking, that sequence configures, builds, then lets call it "outputs" the final organized set of files in a relative subdirectory to the source. Other methods of triggering the "install" all wind up calling the CMake install step, just with the arguments taken from different places. (Even running the "install" project in the VS solution -- in Windows -- just winds up calling CMake as its only action.) Now abrownsword may want to control lthat output a different way--but should still use some equivalent of the steps CMake uses for the "install" step here, at least so that the include directory is cleanly created. (Copying the single pqxx.lib library is not a lot of work, doesn't much matter how one gets at that.)

And (I challenge you) before the "install" step (however done), where is the pqxx.lib library? It's in the src/Debug subdirectory of the "binary" directory I set up above as the intermediate configuration and build files. Was anyone was expecting binary/src/Debug/pqxx.lib? No problem, that is an internal CMake choice--and the "install" step copies it to a location that one would expect: lib/pqxx.lib in the target output location.

[jtv]

@GordonLElliott do I understand correctly then that if you build your own application and libpqxx in one go, using CMake, that the libpqxx build goes through a step that's just like "make install", except it does not actually install the build artefacts to any privileged location on the filesystem?

[GordonLElliott]

Jtv, Not exactly......

Now abrownsword wants to do that -- build application and libpqxx at same time. I am doing an actual "Install" (opposite of abrownsword, what he wants to avoid). Controlling the CMake process from a top level above your's he should be able to automate it just like a script in some manner to combine.

Regarding direct use of your scripts (as I do) one cannot apparently do it all in one step, at minimum two steps running CMake are required, a configure step and then build step. Really the third step, "install" (as discussed above) can be combined sometimes, but in my case it was just my Visual Studio makefiles with a project that called back to CMake install step, so really doing the 3rd step as a 3rd CMake step anyway.

A simpler answer to a slightly different question might be:

YES exactly Jtv! The final step (called "install" in CMake) can simply be directed to a subdirectory relative to your build files (by not referring back to top level directory) and thus is both local to one's working files, but also not in a privileged location. My examples above do that. (And abrownsword could do those steps in a higher level CMake script, making the answer to Jtv's original question yes. Just I'm not doing that, abrownsword is. And to find that "just like", abrownsword or someone will have to find it, or else it will actually be the "install" step, or else will be some completely different method.)

[abrownsword]

Not that I'm aware of. CMake seems to figure out where the built products are put, and uses them directly from those locations when building the application the library will be part of.

[jtv]

@GordonLElliott one of the things you say that keep confusing me is that you talk about multiple cmake invocations. Normally when you build a project using CMake, you only invoke cmake once, and then you use make or ninja or whatever build tool you like for the actual build/install steps, right?

[GordonLElliott]

Jtv, I was going to put this on the installing Windows thread, but since you asked:

I tried to use the documentation (as though naive like back a while). Just CD into the source directory and CMake and it says one has to somehow refer to the source and destination directories (essentially). One of the options it suggests is the out-of-source build option. But mainly the point is without knowing CMake that doesn't work by itself without further options.

The first line below (with the "master" version) will do that first CMake configure step. (Of course replace "master" with the version number on the release source.) Execute the lines in the directory which is above the source code, and the other subdirectory is created in that same top level directory--no muss or fuss.

Yes, after that one can use the Visual Studio tool. But this takes a bit of work--most notably one has to open command window with the proper environment set. And one had to configure any install location back in the first step. Another method at this point is to open the solution with the Visual Studio IDE.

Calling CMake a couple more times is easy, as CMake knows how to set the environment for the tool. In example below the second line does the build step using MSBuild with all the environment so it just runs. I used "binary" as the subdirectory for the configuration/build activity.

Then as we discussed above (but everyone does not seem to agree) the "install" step makes a simple output structure with the include, lib, and "share" directories (that last with documentation).

That last step can be done with the Visual Studio system, but with problems. First it just calls CMake to do the "install" step anyway, so what is the point. Second the default install location is the privileged 32 bit program files directory (wrong place, and fails without permissions anyway).

The list of 3 commands below are simple, even typed into a command window, or as a batch file, and the "install-directory" can be where one wants it (just not a location that requires running as administrator). Add --config Release on the 2nd and 3rd lines to compile and install the Release version (or explicity list --config Debug).

cmake -S libpqxx-master -B binary
cmake --build binary
cmake --install binary --prefix install-directory

Note these are according to top three bullets in: https://cmake.org/cmake/help/latest/manual/cmake.1.html

PS, I assume this will work with other compilers in Windows, and even on other operating systems with similar directory structures--the commands are standard in the CMake documentation.

PPS, that above was going to build the static library. Need flags to build shared library (DLL). (Which I thought was not advisable on Windows, now you are advising that.)

[jtv]

Ah, I see that I left out an important bit that I meant to write in the Configure section. I hope I've filled in the gap. Thanks.

About calling cmake multiple times, I'm not saying it's difficult, I'm just asking: are you meant to call it multiple times? Because I never do. I call cmake once, and it sets up the build directory, and then I can both do both the compiling and the installing using make or ninja. Or is what you're saying that commands like make install or ninja install will just call cmake?

Re-reading your text again, I now think I have a better understanding of what you were trying to say. Is this understanding correct? -> «First thing that happens when you install the project is that CMake sets up a directory with just the files that it wants to install. It would be useful to do just that part, without actually installing.» And then those three cmake commands you give are how you do that, basically by "installing" to an unprivileged directory?

By the way, one tip for making your writing much clearer: Avoid the passive mode in technical writing! Makes a world of difference. With the passive mode I'm always wondering: yes but who or what? It also sets a particular trap in English where "can" or "may" can mean either "it can happen," or "you are allowed to." We normally hear the difference, in the speaker's voice, but not in writing. I keep having to re-read what you're writing in order to resolve these ambiguities.

It also wasn't clear to me what binary meant in your example, so I didn't really know what to do with it. And cmake's man page isn't being very helpful about it.

[KayEss]

cmake -S libpqxx-master -B binary cmake --build binary cmake --install binary --prefix install-directory

The -S option is the source code. The -B option is the directory to place the build in. The following two invocations of cmake are simply using cmake as a wrapper to invoke the build system. On a Unix --build it will call make or ninja depending on how you configured the initial cmake. The advantage here is that cmake will do the right thing no matter what build system it's generated files for. Personally I don't use it either :-)

There are various tools that make configuring cmake easier (ccmake (curses) and cmake-gui on Unix), and it may be worth suggesting their use for people who are not used to cmake -- I don't know what the Windows options are in this space.

[jtv]

Thanks @KayEss — if cmake can act as a front-end for your build tool, that means that I can at least document what commands people should run for building/installing. Right now all I have is "er... depends on your setup I guess."

[GordonLElliott]

I kept fighting with language, (I mean on the actual steps in examples). I used "binary" because it actually agreed with CMake documentation in 2 or 3 places. And I avoided "build" because I was thinking (or writing) that the "build" directory was the top level directory in which everything happened. But I can see that is still confusing. Here I'll go back to "build" as specific intermediate directory and call a top level directory in which everything happens "top level directory". (Maybe that will work).

SO: Put libpqxx-master (or libpqxx-) into a top level directory. Giving commands in sequence:

cmake -S libpqxx-master -B build
cmake --build build
cmake --install build --prefix install-directory
pause

This sequence should work on a larger number of circumstances according to the CMake documentation.

Now I tried the CMake GUI (on Windows) and there are things I can't easily do, rather makes me do even more work. Select the source by point--OK, easy. Then select the destination? Directory has to be there already, and no option to create that empty directory. I find typing the commands easier.

But even more easy is copy the lines, and use programming editor to create a batch file. (Yes can improve on those lines with error checking, but so what.) I'm going to post this note and swipe my own text and do that with latest build and then edit to tell what happens. Just place batch file in top level directory, and click on Windows to execute the batch file.

Result: Edit 1: OOPS, left blank character out on 3rd line so that appears to have failed, and I didn't see result. Trying again, just where I left off not deleting anything just copy again with the edit. Edit 2: That's interesting, build seemd ok but install-directory didn't have any library. I'm adding pause statement (will see 4th line above). Edit 3: Interesting, that failed because the final install tried to install the Release configuration, while the build built the Debug configuration. Will create new sequence below:

cmake -S libpqxx-master -B build
cmake --build build --config Debug
cmake --install build --config Debug --prefix install-directory
pause

And run that: Half an hour later and I have include, lib, and share subdirectories in the "install-directory" subdirectory. To generate I clicked batch file, but could have executed the lines above (less "pause") in a command window. Don't think I would want to use the GUI method, more work than just copying the lines one at a time into a command window.

The interesting sort of failure is that the final "install" step defaulted to Release configuration while all the intermediate steps were Debug. Above steps worked.

Edit 5?: [Remove comment about failure on INSTALL] Turns out CMake in 7.1.2 is just fine, I didn't run it correctly. However I still think the above CMake steps listed are a better approach and don't worry about having Visual Studio have a project that builds the install by calling CMake.

[jtv]

@GordonLElliott before we get back into the install question, could you just answer my question as to whether I understood you correctly? Very important to build a good basic understanding first, otherwise we'll just be building castles in the air.

[GordonLElliott]

@GordonLElliott one of the things you say that keep confusing me is that you talk about multiple cmake invocations. Normally when you build a project using CMake, you only invoke cmake once, and then you use make or ninja or whatever build tool you like for the actual build/install steps, right?

Edit: INSTALL process was working fine in 7.1.2, my problem.

I don't know about "normally", so the answer is NO. Probably different on different operating systems as to how users "normally" use it. But as to whether one could do it that way, yes, not the same as "normally" in my opinion. Now on Windows with Visual Studio, one could then go on with the native tool to do the build and the install steps (once again CMake --build calls native to do build, native "INSTALL" then calls CMake to do the install as set up in configure step).

However in practice it may require setting several variables in the initial configure. In some circumstances may be better off using the 3 steps of CMake. Now there is a pattern used in some CMake builds with the 3 steps. As of CMake 3.15.7 all three steps are available in documented command line options, configure, build, install: https://cmake.org/cmake/help/v3.15/manual/cmake.1.html

Build leaves the user's output in scattered and non-obvious locations (build/src/Debug for the library, and uses original source location of the include--completely separate subdirectory) so I keep pointing out that the "install" step is important even if not placing the library in an operating-system-defined location.

Another point is that at least in the Visual Studio build, the "install" project just refers to a CMake script, it is just a cover for a CMake step anyway for "install".

So as to how those CMake steps work:

1. First (configure) is all CMake, build up the tool's scripting and packaging for all needed projects (theoretically can be done using the tool next). The native tool is chosen in this step.
2. Build step can be triggered by CMake -- but that essentially runs the native tool (chosen in step 1) with proper environment more or less like the user would have run it directly in the native tool. Note that in my case of Microsoft Windows and Visual Studio, I have to run specific batch files or start with specific command windows to get the right environment -- but CMake step 2 does that for you.
3. Install step (once again could just be a local subdirectory organizing the output in our case). When done in the native tool, apparently the native tool calls back to CMake to do the actual "install" job, at least in Windows version. And as of 3.15 CMake has a standardized comand line to do the "install" step directly.

One reason to use the "three step" process is that steps 2 and 3 can be run separately for Debug and Release configurations, whle specifying different final installation locations. Using the build process can be selected for Debug or for Release, for example, in the IDE or by native build tools, but the install output directory remains that single directory set in step 1 and won't distinguish between Release and Debug compiles for example.

[GordonLElliott]

@GordonLElliott do I understand correctly then that if you build your own application and libpqxx in one go, using CMake, that the libpqxx build goes through a step that's just like "make install", except it does not actually install the build artefacts to any privileged location on the filesystem?

(Or this question?) Here abrowsword is doing exactly that sequence--but I suspect he is using the library from the internal location and the include from the original source location, just like the test suite does within the library build. So answer is NO, CMake does not inherently go through an install sequence, or necessarily even build an "install" process.

Somewhere I was suggesting that if the output was then organized in a clean manner (not bundled among the diverse input source and intermediate hard-to-find locations) then something like the "install" process was going to occur--whether using the standard install process that CMake seems to build, or creating an equivalent. That said, the case of the test suite, and the case where abrowsword bundles with a single project, I don't see any problem using the cryptic internal locations (which as he points out CMake knows about). If these are programming internals then no issue. If programmers need ready access to the library includes then I suggest the "install" process (but I don't think that is abrowsword's case).

So in that sense the answer is "Yes". It is possible for the CMake so-callled "install" process to put in a non-privileged location on the filesystem. And I think it is possible (see below for ideas on accomplishing) to call upon the CMake so-called "install" process to do this.

[abrownsword]

It is worth saying here that, IMO, it would be nice if the pqxx cmake process did create a directory structure in its 'output' path (i.e. ${CMAKE_CURRENT_BINARY_DIR}) in which it put all the built products and all the source files (i.e. headers) needed by subsequent use. This is nice for users of a library because it makes them easy for humans to find. The install can then just be a copy command from that directory structure to wherever it is being installed.

[GordonLElliott]

CMake build of libpqxx (normal user invocation) does build a process that does that.

It can be triggered (late versions of CMake) by:

cmake --install build --config Debug --prefix install-directory

and in Visual Studio build project "INSTALL" calling back to cmake, here is the sequence:

setlocal
"C:\Program Files\CMake\bin\cmake.exe" -DBUILD_TYPE=$(Configuration) -P cmake_install.cmake
if %errorlevel% neq 0 goto :cmEnd
:cmEnd
endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
:cmErrorLevel
exit /b %1
:cmDone
if %errorlevel% neq 0 goto :VCEnd

which is executed as a windows batch file. BTW that is 'Microsoft Speak' for

cmake  -DBUILD_TYPE=Debug -P cmake_install.cmake

I would suspect that one could find (and configure and execute) the "cmake_install.cmake" process referred to directly within the CMake scripting. In this case not an "install" but point it to a (third) local subdirectory. If all outputs are configured, will have: include, lib, and a directory with documentation. Obviously if turn off docs build, you get the two desired.

[GordonLElliott]

Thinking about the process again...

The configure step configures a large set of build operations, in the native tool.

Now the example of the test suite is an example that uses the early step, just the compilation of the library, before compiling/linking the test suite. Similar may be the level abrownsword ties in his (your) project.

But still the make is just a series of steps that depend upon others in sequence. The "install" step--not considered installation for users just installation of lib and include in a standardized local place by configuring the "install" step, can be a necessary prerequisite for a later step. Then the later step refers to that library and include there, rather in the intermediate location used by the test suite for example.

Now my attempt in Windows had that step broken, but that can be fixed. Someone needs to test the "install" step to a targeted local directory in Linux/Mac or one or more operating systems. I'll give another go to see if I find what is wrong in my case (must refer to the configuration and location for example in initial setup).

[GordonLElliott]

It is worth saying here that, IMO, it would be nice if the pqxx cmake process did create a directory structure in its 'output' path (i.e. ${CMAKE_CURRENT_BINARY_DIR}) in which it put all the built products and all the source files (i.e. headers) needed by subsequent use. This is nice for users of a library because it makes them easy for humans to find. The install can then just be a copy command from that directory structure to wherever it is being installed.

Ah--there was nothing wrong with the "INSTALL" build process when I thought it was broken (except the nut behind the wheel).

So I suspect the setup to group the lib and include nicely would look like:

However you refer to the libpqxx-<version> source in your CMake scripting.

Define variable: CMAKE_INSTALL_PREFIX=lib-libpqxx wherein lib-libpqxx is a relative directory to your build location. (Choose your own name of course. I don't know how to achieve that from CMake and work specific to the libpqxx CMake scripting, and work on all systems...)

You wanted to turn off documentation and test project build--someone knows how to set variables for that and you have probably already done that.

Then (guys how to do this?) make main project dependent upon the so-called "INSTALL" project of the libpqxx sub-project group.

THEN the location of the libraries will be (relatively) lib: lib-libpqxx/lib/pqxx.lib, include: lib-libpqxx/include/ ... . By the way, that "copy command" is a specific embedded CMake script, somewhere most likely called cmake_install.cmake.

I suspect all this can be achieved and would work almost universally, but I don't know CMake well enough to say for sure, nor how to do some of the steps above.

Also I only observe that working in my Windows/VS environment, but I suspect it works the same on all systems. The "INSTALL" project when "made" (according to methods of the system) calls back to CMake already configured script that was configured by the original libpqxx build and specfiically the CMAKE_INSTALL_PREFIX=<whatever>, wherein that variable meant a (possibly relative) path to the library output (and makes all necessary directory tree in the process).

What I really don't know is in a multi-level project build like this, how to avoid conflicts like "INSTALL" project which is just for the library and not the top level project. Can that be easily renamed (from higher level CMake script) so that it is named "PQXX_INSTALL" for example, avoiding conflicts.

[GordonLElliott]

Note abrownsword has commented on a dislike for "install" processes--at least in his current application.

I just had a spectacular failure creating my own "install" process in Windows for the library--accidentally tried to make my entire "Program Files" directory look just like the library. It started deleting the 'A's first before I killed it, which luckily were all Adobe products (which I easily restored).

[GordonLElliott]

I know it is sort of the opposite of what @abrownsword wanted, but my:

Easy-PQXX Build for Windows Visual Studio has become fairly well developed, and includes explanations of the build process on Windows operating system for Microsoft Visual Studio.

[abrownsword]

It is important that multiple use cases be supported, so having a couple of opposite ones in the documentation is good (as long as they are clearly delineated as alternatives to avoid confusion).

[jtv]

We've had this ticket open forever now, but without actually doing much about it. Would anyone mind if I closed it?

[abrownsword]

Is it resolved?

[jtv]

I don't think so, but it's hard to tell with this sheer wall of text! I'd rather start a new, clear, concise ticket than leave all this sitting around forever.

[abrownsword]

Then I would suggest opening a new such ticket, posting its link here, then close this one. Unless you outright delete this one, readers will find their way here and will want to know where to go when they reach the end of the thread.

[jtv]

I'll start doing that right away then — new tickets first, then retire the old one.

One little piece of progress is that there is now a single, clear home for this kind of documentation: BUILDING-cmake.md.

[jtv]

@abrownsword I just tweaked BUILDING-cmake.md some more. What do you think — does it answer questions like yours now?

[abrownsword]

Looks pretty good. Definitely a big improvement. Thanks.

[jtv]

Glad to hear it. Still have no idea what to write about target properties.

[jtv]

Yeah, I think I'm out of things I can usefully write in there. I think I'll close this ticket and wait for the next one to come along. :-)