[feature] I think we need to add a new package type for source code packages

[M0rtalPhe0nix]

What is your suggestion?

Hello all, in alot of cases produced packages needs to be a source code package that its build has to be defined from the consumer side . currently we tend to use the package_type build_scripts, which for the current time serves the purpose of defining source code packages. but recently we had a bit of a strange scenrio that resulted from our improber use of the build_scripts package type as its default visible trait is false which leads to the unfortunate scenario where we have multipe source code packages with different version in the dependency graph without raising a confilict or resolution of the different requirements of the package to a suitable version . I have attached an illastrative example about this case in the attached image . so according to my current understanding I think an ideal case would be to have native support for source-code packages from conan itself .

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @M0rtalPhe0nix

Thanks for the suggestion.

It would be necessary to step back, probably a lot, to understand the use cases, the problems that are being solved, etc. The recommended way to manage source is not in Conan packages, but in Git repositories, so why putting the source in Conan packages to start with?

What is the meaning of depending on a source package? How are the consumers using that code inside that package?

Also, it is already possible to put code inside Conan packages and use them as regular requires with visible=True (maybe make sure to check headers=False, libs=False, why wouldn't you use them as regular requires? What is the meaning of a "requires" among 2 source packages? In any case, it is very likely that the "source" packages shouldn't be in the "build" context, but in the "host" context, so marking them as "build_scripts" looks indeed not suitable.

[dkoerner-festo]

Hi @M0rtalPhe0nix ,

we also have some cases where we package sources into Conan packages. But we use header-library packages for this.

[memsharded]

Thanks for the feedback @dkoerner-festo

we also have some cases where we package sources into Conan packages. But we use header-library packages for this.

Indeed, header-library packages are "source" packages in this sense, so this would be better than using packages in the "build" context.

If they contain source code .cpp too, is there any pattern or way to automatically use the code in the consumers? Or it requires some custom knowledge that is specific to the consumer? (so it cannot be generalized in some Conan behavior, common for all users)

[dkoerner-festo]

In our case we provide the package folder as CMake variables to the consumer:

In generate():

...
 for require, dependency in self.dependencies.items():
            if dependency.package_folder is not None:
                if require.build:
                    cmake_vars_file_content += f"""set(BUILDTOOLS_{dependency.ref.name.upper()}_ROOT {str(PurePath(dependency.package_folder).as_posix())})
"""
                if dependency.package_type == "header-library":
                    cmake_vars_file_content += f"""set(BUILDTOOLS_{dependency.ref.name.upper()}_ROOT {str(PurePath(dependency.package_folder).as_posix())})
"""
                if dependency.package_type == "build-scripts":
                    cmake_vars_file_content += f"""set(BUILDTOOLS_{dependency.ref.name.upper()}_ROOT {str(PurePath(dependency.package_folder).as_posix())})
"""
...

This works, but it feels more than a workaround, not a final solution.

In the consumer CMakeLists.txt we use this variables to access the sources/header.

[memsharded]

BUILDTOOLS_{dependency.ref.name.upper()}_ROOT

Yes, this is what I mean. The way to use something like this is just a convention in the consumer side, there is not a general way to "consume sources" from a dependency in the same way a library can be consumed with find_package() + target_link_libraries(). So generalizing something like what you have for all Conan users seems difficult, isn't it?

[dkoerner-festo]

At least I haven't found a solution.

[mohamed-omara39]

Hi @memsharded

why putting the source in Conan packages to start with?

What is the meaning of depending on a source package? How are the consumers using that code inside that package?

because these source code packages has a lot of prebuild configurations that the consumer needs to configure before building it. why not use conan file settings and options for this ? you may ask . it is because the number of options can reach thousands and we have an internal tool dedicated for this configuration.

why wouldn't you use them as regular requires

Because in my current setup I need source code packages not to be propagated upstream if it was consumed previously by a binary package (static/shared) libs

What is the meaning of a "requires" among 2 source packages ?

in my case it is a way to allow the first source code package to configure some of the configurations parameters of the second package and allow the consumer of both packages to configure rest of the parameters

[memsharded]

because these source code packages has a lot of prebuild configurations that the consumer needs to configure before building it. why not use conan file settings and options for this ? you may ask . it is because the number of options can reach thousands and we have an internal tool dedicated for this configuration.

Not sure if I understand. Are they source code packages, or do they contain prebuild binaries? Are they actually building library-like artifacts for the package, but those binaries are not being stored or reused, are they always being built from source? Or they are directly built by the consumers of the package, by adding the list of sources to the build targets of the consumer project?

Because in my current setup I need source code packages not to be propagated upstream if it was consumed previously by a binary package (static/shared) libs

You can use something like requires("mysource/version", visible=False), have you tried that?

in my case it is a way to allow the first source code package to configure some of the configurations parameters of the second package and allow the consumer of both packages to configure rest of the parameters

I am afraid that I might be missing a lot of context and understanding of the setup. From the above, it seems the "configuration parameters" are not Conan settings/options, so not sure how this configuration would work or propagate.

[mohamed-omara39]

Are they source code packages?

yes they are . and the prebuild configurations I was talking about is source code configurations are like a group of (#defines) and code behavior configuration .they don't contribute to the build but to the behavior of code itself . and they don't contain any prebuilt binaries .

are they always being built from source? Or they are directly built by the consumers of the package ?

they are directly built by the consumer of the package .

You can use something like requires("mysource/version", visible=False), have you tried that?

the scenario described in this picture happens when I make all my source code package visible=False

To be honest I am on a bit of a predicament where I want to make all the packages that is sources code be visible as long as the consumer is also a source code packages . but if the consumer will build the I don't want to have them propagated upstream . in simpler terms I want the source code packages to be visible upstream until they are built . an ideal scenario for me is if I can check the package_type of the dependencies on requirement function in conanfile to be able to determine the visible trait for each package . or as requested to have a package type that defines this use case .