[question] how to handle a massive change of path to sources in conandata.yml

conan-io / conan

Conan - The open-source C and C++ package manager

https://conan.io

MIT License

8.26k stars 980 forks source link

[question] how to handle a massive change of path to sources in conandata.yml #16861

Open Alexolut opened 2 months ago

Alexolut commented 2 months ago

We have conan recipes with the link to sources stored in our internal repository. Most links stored in conandata.yml files and have a form like https://repo.company-name.com/packet-name/sources.tgz

Repo has been moved to another url like https://new-repo.company-name.com

And now we need to update a lot of conandata.yml files and upload updated recipes to our artifactory server.

The question: what's the preferred list of actions to do in case of migration archives with sources used by conan recipes? Should each package version be updated/changed when the source path is changed?

memsharded commented 2 months ago

Hi @Alexolut

Thanks for your question.

Quick check: wouldn't it be possible to define a redirect in your company-name.com domain? That would be a super quick DNS fix, no need to change anything.

The question: what's the preferred list of actions to do in case of migration archives with sources used by conan recipes? Should each package version be updated/changed when the source path is changed?

This depends on your needs. Doing an automatic replace of urls in conandata.yml is straightforward. The thing is that it will take a while to re-build new binaries for it, but I guess it would be worth. So something like:

Iterate all recipes in your repo. For each one do the url replacements, then iterate the versions in conandata.yml to run a conan export . --version=version to export it with the new recipe revision taking into account the new urls.
Iterate again all recipes, for each version in each conandata.yml, do a conan create . --version=xxx --build=missing (the export above make it work without issues, all has been exported first)

Does this make sense for your case?

Alexolut commented 2 months ago

Hi, @memsharded ! Thank for a quick response.

If I understand you correctly you suggest not to change the versions of the packages, but just do rebuild of them using the updated source location. If so, then it seems that I forgot to mention some restrictions: I don't have direct access to the machine with artifactory, hence I can't do actions recommended by you, and have to upload the packages (recipies and binaries) to the artifactory from the developer's local machine. More over at this time I don't have overwrite permissions (I hope I can request for them for that task, but not sure, so for now I can only upload a new versions of the packages), in this situation I'm not able to replace the old (previously uploaded) packages with the new ones without incrementing the version.

memsharded commented 2 months ago

If so, then it seems that I forgot to mention some restrictions: I don't have direct access to the machine with artifactory, hence I can't do actions recommended by you, and have to upload the packages (recipies and binaries) to the artifactory from the developer's local machine.

This isn't done in Artifactory, this is done in source, in the developer machine. Only the final conan upload needs to be able to upload the packages.

More over at this time I don't have overwrite permissions (I hope I can request for them for that task, but not sure, so for now I can only upload a new versions of the packages), in this situation I'm not able to replace the old (previously uploaded) packages with the new ones without incrementing the version.

This is not an issue. A change in conandata.yml produces a new recipe revision which is a new entity, with its own folder. It doesn't produce an overwrite and it doesn't need overwrite permissions.

Alexolut commented 2 months ago

A change in conandata.yml produces a new recipe revision which is a new entity, with its own folder. It doesn't produce an overwrite and it doesn't need overwrite permissions.

For instance, I have lib/1.2.3@user/channel in my dependencies. When I finish updating the path to source and do upload, the dependency will remain the same. Right? So we will have (I'm not sure how to correctly call it) "detached" binary package on artifictory that is inaccessible anymore. Please, correct me if I wrong.

When I did conan upload lib/1.2.3@user/channel -r our_repo the process stuck due to (I think) absence of overwrite permissions, because if I use new version - everything went smooth.

memsharded commented 2 months ago

For instance, I have lib/1.2.3@user/channel in my dependencies. When I finish updating the path to source and do upload, the dependency will remain the same. Right? So we will have (I'm not sure how to correctly call it) "detached" binary package on artifictory that is inaccessible anymore. Please, correct me if I wrong.

Not really. Every change in source should produce a new recipe revision like lib/version@user/channel#revision1, lib/version@user/channel#revision2 and so on, please read https://docs.conan.io/2/tutorial/consuming_packages/intro_to_versioning.html#revisions. Binaries are always attached to their specific revision that created them, there are no "detached" binaries (they were called "outdated" binaries in Conan 1.X legacy pre-revisions).

If the case is that you are still using legacy Conan 1.X without revisions, the recommendation is to enable revisions, they have been in Conan 1.X for many years now, and they are hardcoded and mandatory in Conan 2.