[question] How to create package with submodule

[kahlenberg]

What is your question?

Hi, I want to create a package from my project using the conan create command. My project includes three submodules. When I run conan install and conan build, everything works fine, and the submodules are fetched from GitHub. However, when I run conan create, I receive the following error: fatal: not a git repository (or any of the parent directories): .git This error indicates that git commands are being executed in the Conan cache directory, which does not contain a .git directory. When I look into cache directory, indeed, there is no .git directory, only CMakeLists.txt and src/ directory.

How can I package my project with submodules?

My conanfile.py:

import os
from conan import ConanFile
from conan.tools.cmake import CMakeToolchain, CMake, cmake_layout, CMakeDeps
from conan.tools.build import check_max_cppstd, check_min_cppstd
from conan.tools.files import copy
from conan.tools.scm import Git
import shutil
class myprojectRecipe(ConanFile):
    name = "myproject"
    version = "1.0"
    package_type = "application"

    # Optional metadata
    license = "<Put the package license here>"
    author = "<Put your name here> <And your email here>"
    url = "<Package recipe repository url here, for issues about the package>"
    description = "<Description of .... package here>"
    topics = ("<Put some tag here>", "<here>", "<and here>")

    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"

    # Sources are located in the same place as this recipe, copy them to the recipe
    exports_sources = "CMakeLists.txt", "src/*"

    def validate(self):
        check_min_cppstd(self, "17")
        check_max_cppstd(self, "20")

    def layout(self):
        cmake_layout(self)

    def generate(self):
        deps = CMakeDeps(self)
        deps.generate()
        tc = CMakeToolchain(self)
        tc.generate()

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

    def package(self):
        cmake = CMake(self)
        cmake.install()
        self.copy("*.dll", dst="bin", keep_path=False)

    def package_info(self):
        self.cpp_info.libs = ["myproject"]

    def deploy(self):
        # Custom deploy to copy only DLLs to the build directory
        build_dir = os.path.join(self.build_folder, "bin")
        if not os.path.exists(build_dir):
            os.makedirs(build_dir)

        for dep in self.deps_cpp_info.deps:
            bin_dir = self.deps_cpp_info[dep].bin_paths[0]
            for dll in os.listdir(bin_dir):
                if dll.endswith(".dll"):
                    shutil.copy(os.path.join(bin_dir, dll), build_dir)   

    def source(self):
        # Cloning the main project repository
        try:
            self.run(f"git rev-parse -C {self.source_folder} --is-inside-work-tree")
            self.output.info("try --------------")
        except:
            project_url = "<ssh git url>"
            self.run(f"git clone --branch conan {project_url} {self.source_folder}")
            self.output.info("except ---------- ")

        # Initialize and update submodules
        self._update_submodule("external/imgui", "https://github.com/ocornut/imgui", "docking")
        self._update_submodule("external/implot", "https://github.com/epezent/implot")
        self._update_submodule("external/imnodes", "https://github.com/Nelarius/imnodes")

    def _update_submodule(self, submodule_path, repo_url, branch="master"):
        if not os.path.exists(submodule_path):
            self.run(f"git submodule add -b {branch} {repo_url} {submodule_path}")
            self.run(f"git -C {self.source_folder} submodule update --init --recursive")
        self.run(f"git -C {submodule_path} fetch")
        self.run(f"git -C {submodule_path} checkout {branch}")
        self.run(f"git -C {submodule_path} pull")

    def requirements(self):
        self.requires("opengl/system")
        self.requires("ffmpeg/6.1")
        self.requires("assimp/5.4.1")
        self.requires("libx264/cci.20220602")        
        self.requires("glfw/3.4")
        self.requires("glew/2.2.0")
        self.requires("freetype/2.13.2")
        self.requires("zlib/1.3.1")
        self.requires("nlohmann_json/3.11.3")
        self.requires("stb/cci.20230920")
        self.requires("glm/cci.20230113")

I am using Windows 10, conan 2.4.0

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @kahlenberg

Thanks for your question.

When creating packages there are 2 strategies to handle the source:

• Do an export sources and do a copy of all the necessary sources from the current local folder to the Conan cache (check https://docs.conan.io/2/tutorial/creating_packages/create_your_first_package.html)
• Use a source() method that retrieves, downloads or git-clone the sources from the original source repo (https://docs.conan.io/2/tutorial/creating_packages/handle_sources_in_packages.html)

You are using a source() method that is only partial, and works only locally, but the source() method should be invariant, complete, and self-contained, it should also work in "isolation" when running in the Conan cache.

You might want to check https://docs.conan.io/2/examples/tools/scm/git/capture_scm/git_capture_scm.html, this is a way of capturing the current repo commit at export time (when copying files to the cache), so the source() method is able to use it to recover the full source code. (if you are doing that, you can remove the exports_sources from your recipe).

[memsharded]

Some other notes:

• You are still using legacy self.copy("*.dll", dst="bin", keep_path=False) helper. Please change it to copy(self, ...) modern alternative
• self.deps_cpp_info is also deprecated and self.dependencies access should be used.
• The deploy() method isn't intended to copy files in the build folder, it should be use to define the deployment of packaged files inside the final package to user locations outside of Conan cache.

[kahlenberg]

Hi @kahlenberg ... You might want to check https://docs.conan.io/2/examples/tools/scm/git/capture_scm/git_capture_scm.html, this is a way of capturing the current repo commit at export time (when copying files to the cache), so the source() method is able to use it to recover the full source code. (if you are doing that, you can remove the exports_sources from your recipe).

Thank you for quick reply. I try to test the method in the link, I receive following error:

        git.checkout_from_conandata_coordinates()
        TypeError: 'NoneType' object is not subscriptable

[memsharded]

Please post the full code, did you add the export() method too?

[kahlenberg]

Yes, I also have export() method. conanfile.py :

import os
from conan import ConanFile
from conan.tools.cmake import CMakeToolchain, CMake, cmake_layout, CMakeDeps
from conan.tools.build import check_max_cppstd, check_min_cppstd
from conan.tools.files import copy
from conan.tools.scm import Git
import shutil
class myprojectRecipe(ConanFile):
    name = "myproject"
    version = "1.0"
    package_type = "application"

    # Optional metadata
    license = "<Put the package license here>"
    author = "<Put your name here> <And your email here>"
    url = "<Package recipe repository url here, for issues about the package>"
    description = "<Description of myproject package here>"
    topics = ("<Put some tag here>", "<here>", "<and here>")

    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"

    # Sources are located in the same place as this recipe, copy them to the recipe
    # exports_sources = "CMakeLists.txt", "src/*"

    def eport(self):
        git = Git(self, self.recipe_folder)
        # save the url and commit in conandata.yml
        git.coordinates_to_conandata()

    def validate(self):
        check_min_cppstd(self, "17")
        check_max_cppstd(self, "20")

    def layout(self):
        cmake_layout(self)

    def generate(self):
        deps = CMakeDeps(self)
        deps.generate()
        tc = CMakeToolchain(self)
        tc.generate()

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

    def package(self):
        cmake = CMake(self)
        cmake.install()
        # to be changed 
        self.copy("*.dll", dst="bin", keep_path=False)

    def package_info(self):
        self.cpp_info.libs = ["myproject"]

    def deploy(self):
        # Custom deploy to copy only DLLs to the build directory
        build_dir = os.path.join(self.build_folder, "bin")
        if not os.path.exists(build_dir):
            os.makedirs(build_dir)

        for dep in self.deps_cpp_info.deps:
            bin_dir = self.deps_cpp_info[dep].bin_paths[0]
            for dll in os.listdir(bin_dir):
                if dll.endswith(".dll"):
                    shutil.copy(os.path.join(bin_dir, dll), build_dir)   

    def source(self):

        git = Git(self)
        git.checkout_from_conandata_coordinates()

    def requirements(self):
        self.requires("opengl/system")
        self.requires("ffmpeg/6.1")
        self.requires("assimp/5.4.1")
        self.requires("libx264/cci.20220602")        
        self.requires("glfw/3.4")
        self.requires("glew/2.2.0")
        self.requires("freetype/2.13.2")
        self.requires("zlib/1.3.1")
        self.requires("nlohmann_json/3.11.3")
        self.requires("stb/cci.20230920")
        self.requires("glm/cci.20230113")

[memsharded]

Sorry, which Conan version? The above is for Conan 2: https://docs.conan.io/2/examples/tools/scm/git/capture_scm/git_capture_scm.html

The coordinates_to_conandata() and checkout_from_conandata_coordinates() only exist in Conan 2, if you want the equivalent in Conan 1 you should do something like in https://docs.conan.io/1/reference/conanfile/tools/scm/git.html#example-implementing-the-scm-feature

Also, you did a typo in the method, you did def eport(self): instead of export.

[kahlenberg]

Sorry, I didn't mention the version. I am using Conan 2.4.0

[memsharded]

Sorry, I didn't mention the version. I am using Conan 2.4.0

then, you will definitely need to update:

• Not use self.copy() it will fail
• Not use self.deps_cpp_info (most likely remove the deploy() method completely.

I think that if you fix the typo it will help.

[kahlenberg]

Also, you did a typo in the method, you did def eport(self): instead of export.

Ok, thank you very much, correcting typo helped and it seems to be working. I will update self.copy and self.deps_cpp_info as well.

[kahlenberg]

One more question: My project contains .gitmodules for submodules, but when I conan create , it doesn't clone the submodules into the related directories. In conan cache, submodule directories are created but empty and project cannot be built.

[memsharded]

The above helpers checkout_from_conandata_coordinates, coordinates_to_conandata are for a single repo storage and checkout. If you have submodules you will need to detail in your export() and source() methods what you want. The Git https://docs.conan.io/2/reference/tools/scm/git.html helper has some methods that you can use, but it is basically a thin wrapper over the git cli tool, so you can also do any self.run("git ...") command easily.

It is possible that you only need to modify the source() to force the cloning of submodules, the implementation of checkout_from_conandata_coordinates is

    def checkout_from_conandata_coordinates(self):
        """
        Reads the "scm" field from the ``conandata.yml``, that must contain at least "url" and
        "commit" and then do a ``clone(url, target=".")`` followed by a ``checkout(commit)``.
        """
        sources = self._conanfile.conan_data["scm"]
        self.clone(url=sources["url"], target=".")
        self.checkout(commit=sources["commit"])

So doing those lines yourself and maybe passing extra args in self.clone() is enough.