[question] Registering user-created conanfile.py for dependencies in main conanfile.py

[maichmueller]

What is your question?

Hi,

In my current project I have a main conanfile.py at the root level. I also defined 3 conanfile.py files for dependencies in subfolders project/dependencies/dep1|2|3. I then stitched it all together with a configure script that has a segment like this:

conan create dependencies/dep1 --build=$policy (mostly 'missing')
conan create dependencies/dep2 --build=$policy 
conan create dependencies/dep3 --build=$policy 
# main project conanfile now
conan install . -of=build/conan --build=$policy 

I create these packages actually only so that the conanfile.py of those dependencies is registered for this library name (I don't know if this is stable behaviour in fact or whether this might change in the future).

What I would rather do: Register the conanfile.py of each dependency so that conan knows about it (without having to build create the entire package!) and let the main conanfile.py then build those deps when told to do so by the --build policy.

This would help me in two ways:

1. I don't have to replicate which version of this dependency should be created as a package (once in defined in the main conandata.yml and a second time then in my glue-script)
2. In a CIBUILDWHEEL step one has to build everything from source to not get wrong glibc linkage etc. In those cases I am rebuilding costly dependencies many times over instead of only once! (e.g. dep1-3 depends on boost each, so each has to build boost to create their respective package. My main project also depends on boost and dep1-3, so i will build boost from source 4 more times. By the end I will have built boost 7 times...

In summary, my (actually 2) questions are

1. Is there a smarter --build policy for the CIBUILDWHEEL case to not rebuild already built dependencies by force?
2. Can I register conanfiles in conan?

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @maichmueller

thanks for your question

I think you are looking for the "editable" packages. You want to put your dependencies as editable, so your consumer project finds them locally and not in the Conan cache. You can even do an orchestrated build with --build=editable (over the consumer app)

Please have a look at https://docs.conan.io/2/tutorial/developing_packages/editable_packages.html and let us know

[maichmueller]

Hi @memsharded,

thanks for the info! An editable package is not quite my intention.

For question 2) Consider a 3rd-party lib (that I don't develop) for which no conanfile recipes exist yet, so I make my own conanfile.py that configures this lib (such as the nauty lib example of my previous issue #17193 ). My project proj now requires nauty/2.8.8, so I call conan create <nauty_conanfile> prior to calling conan install <proj_conanfile>, in order to let conan know about <nauty_conanfile>. In fact, I do this on demand in a glue-script:

conan create <nauty_conanfile> conan install <proj_conanfile> --settings=...

Since the settings will decide the binary compatibility of the just built nauty package, I will probably simply trigger another build of nauty right after. This is pointless, since all I wanted was for conan to know where is to build it when required by another project.

--> How do I register without calling conan create <nauty_conanfile>?

For question 1) The rebuilding problem becomes worse if the --build policy is "*". The use case is that for building python wheels of my python extension proj, one needs to build ALL dependencies from source in a manylinux container. Now consider the requires scenario of proj:

└── proj
    ├── boost/1.86
    ├── dep1/*
    │   └── boost/1.86
    ├── dep2/*
    │   └── boost/1.86
    └── dep3/*
        └── boost/1.86

This means that "*" will build boost 4 times, but it should only build it once and then reuse that build in the other dependencies as well. Note that I cannot use --build=missing, because that will simply download boost binaries which are not compatible with the container's glibc version.

--> So I am looking for e.g. a build option that says "build from source only those libraries that have not already been built"

[memsharded]

Oh, I think I understand now what you mean with register a conanfile.py

This will be the conan export <path-to-conanfile.py> command. That puts the recipe only in the Conan cache, in a very fast way, just milliseconds, making it available for other packages to consume (and build from source when requested with --build=missing), but without really "creating" or building the binary at that time

For conan-center-index github forks, that is, third party dependencies-like, you might even use the local-recipes-index feature, see https://docs.conan.io/2/devops/devops_local_recipes_index.html, which basically automatically exports the recipes on demand when needed, directly from your folder.

[maichmueller]

Ah perfect, that does look like what I want. But Is there a way to export all versions defined in an adjacent conandata.yml?

A local recipe index is probably just right for my use case since then I could even reuse the recipes between CIBUILDWHEEL python version builds. I will try this out and get back to you! Thanks a ton for your help @memsharded !

[memsharded]

But Is there a way to export all versions defined in an adjacent conandata.yml?

Not as built-in, but it might be relatively easy to implement your own custom command that does it. As a reference you have here a "export-all-version" command we use for ConanCenter: https://github.com/conan-io/conan-extensions/blob/main/extensions/commands/cci/cmd_export_all_versions.py. You can adapt it, to limit to some packages and not the whole ConanCenter

But yes, the local-recipes-index might help too, working as "lazy", with recipes being exported on demand when they are needed, for the versions that are needed.

Happy to help!