[question] How to distribute different content to different consumers

[maitrey]

What is your question?

Dear Conan Folks,

We have a use case for example a component has a simulink model , we generate code at build time and compile it , generate the executable package it in 3 different variants(3 different compilers). Now we have a requirement:

• Some customer of us , wants the model but other customers should not have access to the model.
• Some customers want to have the executable(thats easy take it from the artifactory) What would be the right way to proceed:
  1. One option that I thought is package the model only as a variant but this leads to unnecessary building and testing of package. Additionally, artifactory does not have fine grained access to packages meaning that you have either access to the artifactory repository or not.
  2. Second option that I thought is every customer shall have its own remote. When we package the compoennt we package the model, executable and the header. We create then another package from the contents of this package and re-package the model and upload it to another artifactory repository where customer has access and can get his model. This could be tricky as if package has many package revisions how do we map what we distribute? Is there a better way to handle such a use-case?

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @maitrey

First a quick question. When you say "the model", is this the sources used to build the package binary from source? Does the model live in a git repository?

[maitrey]

The model is a simulink model and yes it lives in a git repository.

[memsharded]

Thanks for the info.

Then the access to the model best approach could be via the Git scm model. Not adding the sources as exports_sources, but just capturing the url+ commit information: https://docs.conan.io/2/examples/tools/scm/git/capture_scm/git_capture_scm.html#examples-tools-scm-git-capture

This way, the access to the source code is done via the standard access to the git repo. Those with access to the git repo can build from source, and see the source, those without access to the git repo will only be able to use the pre-compiled binaries exclusively, and if they try to build from source, it will fail telling them they don't have permissions to the model source repo.

Then, to provide different level of access to different customers to different packages, definitely the recommended way is setting up a different repository for each customer.

[maitrey]

We donot want to give customers access to our repository. We wanted to manage it via artifactory as a distribution platform. :(

[memsharded]

We donot want to give customers access to our repository. We wanted to manage it via artifactory as a distribution platform. :(

But this is what I am saying, setting up different Artifactory repositories, one for each customer, in Artifactory, with different permissions, and distribute via Artifactory. So you don't have to give access to your main Artifactory repositories to the customers, just to their own dedicated ones. I think there is something that I might not be understanding.

[maitrey]

Then we are on the same page but question is in order to repackage the content belonging to each customer artifactory repo , I need to take care of mapping PREV with RREV incase if there are multiple versions of the same package. Also for traceability reasons, I need to additional information to the re-packaged content , is my understanding right?

[memsharded]

is in order to repackage the content belonging to each customer artifactory repo

But no need to re-package, you can just copy packages from your own repository to the customer repository. If you have already the package and it is working fine, why creating another? There are still things that I don't understand.

I need to take care of mapping PREV with RREV incase if there are multiple versions of the same package. Also for traceability reasons, I need to additional information to the re-packaged content , is my understanding right?

Sorry, I don't know what you mean. Yes, you need to copy to your customers repositories the packages you want, the versions you want with the revisions you want, like every release process, you don't release every commit, you pick one and make a release out of it, same but with package binaries, isn't it?

Also for traceability reasons, I need to additional information to the re-packaged content , is my understanding right?

You mean adding properties in Artifactory? what "additional information"?

[maitrey]

James you meant copy packages and I meant makinga recipe and downloading it and repackage like : https://docs.conan.io/1/creating_packages/existing_binaries.html And when I follow the above method it could be that I need to take care of adding the PREV and RREV. For example properties like git tags, which variant etc.. customer could subscribe to one variant but not the other ...

[memsharded]

Do you mean that you use conan export-pkg to create packages? Why is that re-packaging? the docs you linked are for creating a package for an existing binary, not for another Conan package. How is it related to distributing different content to different customers in server repositories? The way packages are created is not really related to how different server repositories will give access to different customers to the contents of the repo.

And when I follow the above method it could be that I need to take care of adding the PREV and RREV.

Adding the prev and rrev where? Those are built-in in Conan, no need to add them anywhere?

For example properties like git tags, which variant etc.. customer could subscribe to one variant but not the other ...

The git tag of the original source to build the package is intrinsic part of the recipe, typically the conandata.yml, otherwise the package cannot be built at all. The variant if it is the package binary identified by a package_id, that is also built-in and core, no need to add it anywhere?

I am afraid that I don't understand neither the big picture nor the implementation, the naming/nomenclature used, the flows...

[maitrey]

''' class HelloConan(ConanFile): name = "hello" version = "0.1" settings = "os", "compiler", "build_type", "arch"

def build(self):
    if self.settings.os == "Windows" and self.settings.compiler == "Visual Studio":
        url = ("https://<someurl>/downloads/hello_binary%s_%s.zip"
               % (str(self.settings.compiler.version), str(self.settings.build_type)))
    elif ...:
        url = ...
    else:
        raise Exception("Binary does not exist for these settings")
    tools.get(url)

def package(self):
    self.copy("*") # assume package as-is, but you can also copy specific files or rearrange

def package_info(self):  # still very useful for package consumers
    self.cpp_info.libs = ["hello"]

''' Instead of the zip use the conan_package.tgz and copy required variant binary/header for each artifactory customer repo. So for example I have a package A in X repository , lets say in 3 variants. I would use the above recipe to download package A from X repository and copy the needed variant as package B in Y repository. In order to trace it, I need to add properties like which variant I added , version name would be same.

[memsharded]

Instead of the zip use the conan_package.tgz and copy required variant binary/header for each artifactory customer repo.

So are you hardcoding the full Artifactory URL down to the conan_package.tgz? I am afraid that this is not a recommended practice, you are abusing a lot of internal implementation details that can break at any time, and you also have to hardcode the package_id, recipe_revision and all of those things into the URL, instead of having the Conan recommended resolution, based on profiles, etc. Not to say that dependency resolution is none, so anything that will have any dependency like shared libraries or whatever will be extra complicated to manage.

So it seems we are not talking really about different repositories in the server side, but about creating new packages from existing ones while breaking the dependency chain, for final distribution. It seems the ticket you want to track is this one: https://github.com/conan-io/conan/issues/13171

To extract binaries from Conan there are other recommended and documented mechanisms, like deployers, doing copy() in generate(), etc. I would recommend at this point a 2 step process:

• conan install ... --deployer=mydeployer to colllect binaries from a dependency graph
• conan export-pkg ... to create the new binary from the collected binaries

Those, provided the right and common profiles, will do the task without needing to worry about the details of the URLs or hardcoding anything. The deployer can automate a lot of the data collection and generation of files for the final package.

[maitrey]

Okay james. Sorry, it took quiet some iteration to exactly explain what I had in mind(for that atleast you will remmeber me :D , I hope). Thanks for your inputs. I will try this and get back to you.

[maitrey]

To begin with , I tried to use full_deploy and direct_deploy. Both need a conanfile.txt. Is my assumption correct ?

[valgur]

The CCI imgui recipe packages its different backends as source files, which sounds somewhat similar to your use case: https://github.com/conan-io/conan-center-index/blob/8305ea0ec1be297f700d10531682332339e8d14e/recipes/imgui/all/conanfile.py#L73-L79 An example of a recipe consuming these: https://github.com/valgur/cupoch/blob/9bd39ef5bb77f20583d085bfaeaff6e5830245d3/conanfile.py#L171-L186

[memsharded]

To begin with , I tried to use full_deploy and direct_deploy. Both need a conanfile.txt. Is my assumption correct ?

No, deployers can also work with any conanfile, also .py, and even without conanfile at all, but using --requires= argument.

[memsharded]

The "vendor" feature has been merged in https://github.com/conan-io/conan/pull/16073, will be released in next Conan 2.4.

I think that complements well the alternatives cited above, and it could be a better fit than the deployers for actual later redistribution as a Conan package too, which seemed the original intention.

So this issue can be closed, please create new tickets against the new Conan 2.4 "vendor" feature if necessary (or about the other alternatives like deployers), many thanks!