[suggestion] Ability to declare that you need the dependency source, not binary

[DavidZemon]

Very similar to #719, but the solution of go-style packages is not sufficient. Consider the following...

For embedded systems (think small microcontrollers with 32kB of RAM, not Raspberry Pi level of "embedded systems"), things get compiled very "special" compared to the non-embedded world. To get the smallest possible code size, the developer might want to play around with a variety of different compiler flags - most commonly, -ffunction-sections and -fdata-sections. This could be accomplished by providing a CMake toolchain file with those flags, but they're really not the kind of flags that belong in a toolchain file (they're not CPU-specific, like -march=..., for instance). What would be better, and more common for embedded systems, would be to just include the source of the dependency in the main project and build it with all the same flags as the rest of the project. A git submodule, for instance, would do the trick nicely. But I like Conan... and I'm already using Conan... do I really need both submodules and Conan? Can't I have an all-in-one tool?

Okay, so far, creating a Conan package that copies from source to package folder would, in fact, be sufficient. Until I throw in one extra requirement: I also want to use this library for non-embedded development. In the non-embedded world, I expect to be able to use find_package(linenoise REQUIRED) and target_link_libraries(foobar PUBLIC linenoise) just like every other package and library in my project. I want library binaries when building for Linux, and I want source when building for embedded systems.

Right now, solving this problem requires a Conan option that chooses between compiling and installing binaries or just copying source. But that requires the library author thinking ahead enough to write the correct recipe (or me creating a custom recipe). It would be awesome if I could add source_requires = linenoise/3.0.3@foo/bar and Conan would provide my CMake build system with a variable that points to the project's source root. No one's recipe would need to be changed to enable this feature because of course Conan already knows how to acquire the source (if someone were to be stupid enough to do this on a package that had neither scm, exports_sources, or def source(), Conan would of course just provide a path to an empty directory, or maybe throw an exception.. probably empty directory).

Understood? Clear as mud?

Simple example library where I'd like to be able to use it both from Linux and embedded system: https://github.com/antirez/linenoise

[d135-1r43]

This feature is very interesting. Would the Conan team consider a pull request for this feature if I would work on it?

[memsharded]

Hi @DavidZemon @d135-1r43

Sorry this issue fell off the cracks long time ago. I would like to understand better the feature, from the description it is still not fully clear to me. What is possible is to have pure src packages, lets say "mypkg-src/version@user/channel", that can be required by other packages, and it can be required by "mypkg/version@user/channel" as an origin of the sources. Would this be useful?

If you could please elaborate a bit more how the solution is envisioned from the user point of view (fake recipes for example), and how that solves the problem, that would be very useful. Thanks!

[DavidZemon]

Let's use a simple example... a basic library called string-utils/1.0.0@foo/bar. It's a C++ library that adds high-level string functionality like you might see in Python str class to C++ std::string. This code is applicable to a wide array of platforms, from Linux and Windows to small 8-bit microcontrollers. For x86 Windows and Linux, it makes sense that binaries are stored on a server for common architectures, and that a shared or static library is built on-demand based on standard compiler flags configured in string-utils's Conan recipe for any architecture not in the common binary repo.

However, on my STM32 microcontroller - or any other bare-metal embedded system - it is more likely that I want COMPLETE control over the compiler flags. I could be using various combinations of -ffunction-sections or -Osize or who knows what special compiler flags to make this library work just the way I need on my special snowflake. To do all of this, I don't need or want a second string-utils recipe (perhaps I'm not the original author of string-utils, which would imply forking the upstream repo and creating my own recipe... not ideal). Instead, I'd rather have an option in Conan that just says "give me path to the source for string-utils/1.0.0@foo/bar". Conan has access to this data via the recipe (of course, an error can/should be thrown if the recipe does not provide source info) so it doesn't seem like a big ask.

Btw, another way to look at this is: i'm asking Conan to act as a replacement for git submodules.

[jgsogo]

This is how I see this feature... don't take it as an upvote or downvote.

We need to think about this example in the context of a graph. Nothing can be added/required in the recipe, neither in the consumers (there could be transitive dependencies we want to consume as sources and we don't want to modify recipes in the middle). So the only way to implement this is by using an opt-in, a flag in the conan install command, or some item in the consumer.txt file that will activate this behavior.

Then all the graph will be evaluated by Conan and, using this special behavior, Conan will execute the source method for all the packages in the graph and stop (it is like --build but without building). Then similar to a generator would run taking the paths to all those sources and copying them to the local folder or just providing a path to those folders in the cache (I feel more comfortable with this second one).

Abusing notation (because a generator cannot access this information) it could be:

conan install boost/1.72.0@ -g harvest_sources[cmake] --profile xxxx

and after a couple of minutes (ey, it has to download several Mbs) Conan will provide a file like:

conan-graph-sources.cmake

set(BOOST_SOURCE_DIR .conan/data/boost/1.72.0/_/_/source)
set(BZIP2_SOURCE_DIR .conan/data/bzip2/x.y.z/_/_/source)
set(ZLIB_SOURCE_DIR .conan/data/zlib/1.2.11/_/_/source)

Then it is up to the user to consume that information: run a GLOB and add everything to the application target, copy sources and patch them,...

[DavidZemon]

@jgsogo, I think that would work, but wouldn't it also build each of the dependencies unnecessarily when I go to run conan build ...?

[jgsogo]

Mine was just a conceptualization. If Conan is responsible to build only the root of the graph, something else is needed. Not sure if that could be achieved without a more powerful graph model.

[memsharded]

Please @solvingj feedback here

[solvingj]

We have several historical tickets relating to the propagation of sources and source-related information from producer to consumer packages. Many of these are not completely covered by imports() and also include the specific desire for support in generators. I have implemented source-packages myself in conan and written custom generators to make use of them, so I have some real experience with this. I will try to link this ticket with some of the others, probably into a new "parent" ticket, derive the commonality, and hopefully begin working on some proof-of-concept stuff in the next week or two. Please stay tuned, thanks for the feedback @DavidZemon

[d135-1r43]

The comparison to SVN externals or GIT submodules is very good. The use case:

There is some C code in repository R1 that is dependent on code in R2. Instead of building a library from the code in R2 and using that binary in R1, R2 is added to R1 as a git submodule. So the code in R1 is available at compile time. We want to get rid of the submodules and want to use Conan for dependency management.

The feature as I see it:

A requirement can be defined as a code-only requirement in conanfile.py. Conan will resolve the dependency tree for all code-only requirements and take care, that the code of all transient requirements is available at compilation time -- just at it is when using git submodules.

Some rough example:

def build_requirements(self):
    self.build_requires_code("embeddedmathlib/0.1.0@user/stable", "mathlib/")

At compilation time, the code of embeddedmathlib 0.1.0 is available in the folder mathlib/

[memsharded]

Hi @d135-1r43

I think the above scenario is very easily implementable with existing requires and doing a simple copy of the sources, which is 1 line of code. A small (a bit outdated, but the idea is there) proof of concept is here: https://github.com/memsharded/source_packages

The source package is very straigthforward, just package the sources:

class SourcesConan(ConanFile):
    exports_sources = "src/*"

    def package(self):
        self.copy("*")

The consumers only need to copy those sources in their build()

class ConsumerConan(ConanFile):
    ...
    # maybe use better regular requires not build_requires if we need to affect the ID
    build_requires = "source/0.1@user/testing"

    def build(self):
        src_folder = os.path.join(self.deps_cpp_info["source"].rootpath, "src")
        shutil.copytree(src_folder, "src/external")
        cmake = CMake(self)
        cmake.configure(source_folder="src")

Please have a look at this, as I see it, this implements this use case, in the scenario when we want to actually create "source packages".

A different story is when we want to have a copy of the sources of a package that is actually create binaries and not packaging the sources. Even if there is a package that create binaries for other platforms, it is possible to add an option, like options = {"source": [True, False]} and default it to False, and use it to also create a package with the sources, that would behave as the example above.

And finally, it seems the use case requested by @DavidZemon is when we have the package that is creating binaries for other platforms, but we are not packaging the sources at all. The challenge here is that the process of obtaining the sources is still tied to a given package binary build, with a certain configuration. There is no "give me the "pure" sources" abstractions, because depending on the package, the sources might be different for different platforms, have patches. We will be investigating this, but it will certainly require more work and time.

Would the above work for your use case?

[memsharded]

Conan 2 provides the new conf:

tools.build:download_source: Force download of sources for every package

That allows even a conan graph info to force the download of sources for dependencies, and reporting them in the graph json output:

conan install --requires=zlib/1.3 -c tools.build:download_source=True --format=json

Closing as solved