[question] How to package efficiently

[diogomatos3]

What is your question?

Hi community,

I have a repo, which produces several libraries, that are used by other repos. Lets assume:

• Lib1
• Lib2
• Lib3
• Lib4

Initially, I used package

• package to gather to gather the header files
• package_info to set the cpp_info, so that cmake config files.

My concern and doubt is if the repo continues to grow, conan recipe gets bigger and some team members still like to run cmake commands.

What would be the recommended way to go, on packaging into conan package? Is it reasonable to have on each cmakelist of each component, the respective cmake install and export commands and with conan just handle the packaging into conan? Or in the conan recipe shall be possible to handle such use-case?

Thanks in advance!

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @diogomatos3

Thanks for your question.

I think it heavily depends on the use case and the pains you are trying to solve with Conan packages. There are some different possible aspects:

• The "consuming" aspect: How are those libraries consumed? How large are the binaries for those libraries? Can users use version 1.0 of lib1 and version 2.3 of lib2 and so on? or they must use all of them under the same version. Are there some libs which are very heavily used and others very rarely used? How many other Conan packages are being consumed by those users, besides this repo ones?
• The "evolution" aspect: Is this repo really planned to grow fast? Is it planned to grow very large? How it will be managed in that case (because large mono-repos have their own challenges)?
• The "creation" aspect: How are those "libs" inside the repo actually developed and released?

There are 2 opposed development paradigms: using monorepo builds (live at head) and package-based development. It is fine for a certain repo to create more than one library, and it can still be modeled into a Conan package with components defined. But that is different from having the main development flow to be mono-repo based and pretending to also make packages out of the different mono-repo subfolders at the same time, because those opposite paradigms will collide.

Typically if you don't want to split the mono-repo into different independent repos, with their own development flow, it is also typically not well suited to try to make Conan packages from that repo, and the easiest is to use 1 single Conan package with components. But that depends on the above aspects.

For occasional development of more than 1 package simultaneously the Conan editables feature might help, but it will never provide the same integrated experience as working with one project/build of a repo with different sub-folders.

Please let me know if this helps.

[diogomatos3]

Hi @memsharded

Thanks for raising the questions, it helps to look deeper into the use-case.

The "consuming" aspect: - How are those libraries consumed? . those libraries are just some logic; they are linked into a targets implemented on another repository. - How large are the binaries for those libraries? . very small libraries - Can users use version 1.0 of lib1 and version 2.3 of lib2 and so on? or they must use all of them under the same version. . they use all of them under the same version - Are there some libs which are very heavily used and others very rarely used? . within these set of libs, yes i believe there is different levels of usage - How many other Conan packages are being consumed by those users, besides this repo ones? . few other conan packages from the conan center and from other department

The "evolution" aspect: - Is this repo really planned to grow fast? . at the beginning yes, but then it will stabilize - Is it planned to grow very large? . difficult to answer, i expect few more libraries (lets assume 10 libs in total) - How it will be managed in that case (because large mono-repos have their own challenges)? . at the moment, we were considering maintaining these libraries in the same repository

The "creation" aspect: - How are those "libs" inside the repo actually developed and released? . at the moment, these libs are a subdirectory on the future consumer repo. So, a huge monorepo. The next goal would be to "move out" the libs, and "stablish the link" through conan. The idea is that who works on libs, most of the time doesn't need to be exposed and handle the height of the main repo.

[memsharded]

Thanks for the feedback.

I'd say that at the moment, creating 1 single Conan package out of that repo, that contains the different libraries inside, modeled as self.cpp_info.components would be a reasonable trade-off between the different approaches pros and cons.

. at the moment, these libs are a subdirectory on the future consumer repo. So, a huge monorepo. The next goal would be to "move out" the libs, and "stablish the link" through conan. The idea is that who works on libs, most of the time doesn't need to be exposed and handle the height of the main repo.

Depending on how the rest of the huge mono-repo works, this might be a necessary first step before making a Conan package out of it. It is possible to make a Conan package from the mono-repo build too, but it is not very straightforward to use the result of that package in the mono-repo build itself. But if the package is to be consumed externally by other projects (not the mono-repo one), then it might be feasible to create the package from the mono-repo without extracting it first.

[diogomatos3]

True, we foresee that the same package of libs might be useful on another repositories. Due to supporting different OS systems, which then have different software stacks.

As of today on proof of concept, I have for "packaging":

• in the lib cmakelist, at the bottom, after defining the target type and its dependencies

#-----------------------------------------------------------
#- Installing target and header files

#make cache variables for install destinations
include(GNUInstallDirs)

#install the target and create export-set
install(TARGETS ${LIBRARY_NAME}
        EXPORT ${LIBRARY_NAME}_Targets
        ARCHIVE COMPONENT ${LIBRARY_NAME}
        INCLUDES DESTINATION ${LIBRARY_NAME})

#install header files
install(FILES 
  ${CMAKE_CURRENT_SOURCE_DIR}/include/header_file.hpp 
  DESTINATION 
  ${CMAKE_INSTALL_INCLUDEDIR}/${LIBRARY_NAME})

#-----------------------------------------------------------
#- Generate and install export file
install(EXPORT ${LIBRARY_NAME}_Targets
        FILE ${LIBRARY_NAME}_Targets.cmake
        NAMESPACE ${LIBRARY_NAME}::
        DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${LIBRARY_NAME}
)

When triggering cmake.install() on conan, I see the binaries and headers properly collected, into package folder. But to actually packaging into conan package, so that it can be uploaded, I would still have to implement package_info, using self.cpp_info.components, right?

One of my goals was still be able to maintain the possibility of

1. runing cmake commands on terminal, for developers use case and debug on a certain library with vscode, for example.
2. being able to run conan commands for CI automation and integration purposes.

[memsharded]

But to actually packaging into conan package, so that it can be uploaded, I would still have to implement package_info, using self.cpp_info.components, right?

Yes, in the general case, the recommendation is to fill the package_info() and define the components and different information for different consumers to work properly. However, it might be possible to use CMake generated xxxx-config.cmake files if they are packaged (see https://docs.conan.io/2/examples/tools/cmake/cmake_toolchain/use_package_config_cmake.html), but that might have some limitations, read the docs to know them.

runing cmake commands on terminal, for developers use case and debug on a certain library with vscode, for example.

This works pretty nice with the VSCode cmake-tools plugin, as it recognizes CMake presets, and conan install generates CMake presets. This is a bit independent of the "packaging" approach, here we are talking about the "consuming packages approach".