[question] incremental upload

[Peddaahh]

What is your question?

I have a question to the conan upload. I am building the conanpackage with jenkins in a matrix build for different archs. so every matrix branch, is uploading the resulting artifact on their own (can be even on different machines). Currently with just upload, they are the same package (name) but different revisions on the server. If I now for example, want to install it, it only finds the last revision which has only one architecture included. The others are previous revisions which are not found by conan install.

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[AbrilRBS]

Hi!

This might be because of differing newlines between OSes, check https://docs.conan.io/2/knowledge/faq.html#error-obtaining-different-revisions-in-linux-and-windows for more info and let me know if that fixes the issues you are facing :)

[Peddaahh]

hm no it does not fix my issue, maybe i clarify it more so i am building e.g.: pkg/1.0 and in the matrix build there is the same package name and version built, but the settings.arch differs everytime. so if I upload it now, instead of combining the package which in my head should then look like this:

conan-other
  pkg
    pkg/1.0
      revisions
        abcdef
          packages
            abcdPACKAGEID1def
              info
                options
                  arch: win32
         abcdPACKAGEID2def
              info
                options
                  arch: win64

it is just

conan-other
  pkg
    pkg/1.0
      revisions
        abcdef
          packages
            abcdPACKAGEID1def
              info
                options
                  arch: win32
        ghij
          packages
            abcdPACKAGEID2def
              info
                options
                  arch: win64

so instead of split in packages within one revision, there is a new revision being added which i do not want. i want the packages to be within one revision

[AbrilRBS]

I see! Then the issue might arise from both configurations containing different sources than expected.

If you run conan cache path pkg/1.0#abcdef:abcdPACKAGEID1def, Conan will spit the local folder for your built package. In this folder, a file called conanmanifest.txt is located, which lists all the files that Conan is using to generate the recipe revision.

Could you try running the command for both revision:packageid and check the contents of the manifest. Does it differ unexpectedly?

[Peddaahh]

well

[memsharded]

@Peddaahh those are the package-revisions not the recipe revisions.

The recipe revisions you are showing above abcdef, will be the conanmanifest.txt that is located in the recipe folder.

Sorry there were some error in the recommendation above, it will be the paths in:

conan cache path pkg/1.0#abcdef
conan cache path pkg/1.0#ghij

Excluding the package_id last part. That will give you the manifests of the "recipe", that define the recipe revisions abcdef and ghij. Can you please try that instead?

[Peddaahh]

So Conanfile is the same.. as it runs on different systems, the venv differs heavily, but that can be excluded in export sources... also some .git files differ, as it is pulled everytime on it's own in the matrix build and of course the built package differs, as it it build externally (LabVIEW Builder)

[memsharded]

Ok, that is the reason: you should not be exporting all those private files, both the .conan cache files and the .git files are problematic. Only the specific files that the recipe needs have to be exported.

Please change the pattern to be more specific, like exports_sources = "src/*.cpp" or something like that, and try again.

[Peddaahh]

hm... but the git repo is fetched externally by jenkins.. and then i want to input it into the conan build process to run

    def set_name(self):
        repo = subprocess.getoutput('git config --get remote.origin.url')
        self.name = subprocess.getoutput('basename -s .git ' + repo)

    def set_version(self):
        _ver = subprocess.getoutput('git describe --tags --always').split('-')
        if len(_ver) == 3:
            self.version = f"{_ver[0]}.{_ver[1]}-{_ver[2]}"
        else:
            self.version = "".join(_ver)

[memsharded]

You can get the name and version from the repo, because that is executed before the export.

But you shouldn't be copying all of those files inside the recipe, that is what is happening, it is not necessary for set_name() and set_version() to run. You are basically copying the Conan cache files and the internal .git files inside the Conan recipe, that should be avoided.

[Peddaahh]

Oh okay.. thanks 👍🏻 Is there a way to copy everything except, those folders, or do i have to define every file/folder by itself? Would look like this

    def export_sources(self):
        copy(self, "README.md", self.recipe_folder, self.export_sources_folder)
        copy(self, "*.projj", self.recipe_folder, self.export_sources_folder)
        copy(self, "_LV*", self.recipe_folder, self.export_sources_folder)
        copy(self, "tools/*", self.recipe_folder, self.export_sources_folder)
        copy(self, "src/*", self.recipe_folder, self.export_sources_folder)
        copy(self, "documentation/*", self.recipe_folder, self.export_sources_folder)
        copy(self, "builder/*", self.recipe_folder, self.export_sources_folder)

[AbrilRBS]

Hi @Peddaahh sorry for the misleading initial command recommendation, I misstyped!

Is there a way to copy everything except, those folders

Yes, the copy method has an exclude argument, a list of patterns of elements not to copy (Docs here)

[Peddaahh]

Okay, yeah but how do i cope with it, if the project is built externally, and then packaged into a conan package, therfore my binary that is being packaged differs everytime, as it is built for different arch

[memsharded]

Okay, yeah but how do i cope with it, if the project is built externally, and then packaged into a conan package, therfore my binary that is being packaged differs everytime, as it is built for different arch

Yes, the binary can and will be different for every different architecture (or any binary variant). But the recipe-revision is not the revision of the binaries. It is the revision of the recipe, which most of the times will be the conanfile.py + exported sources. These exported sources must be the same for all binaries, in the same way you can build different binaries from the same git commit. The problem is that you are exporting as sources many other files that shouldn't be sources of the recipe.

[Peddaahh]

But the exported sources, include my binary, which is built beforehand by Jenkins.. therefore it differs everytime..

[memsharded]

No, you cannot export-sources binaries, specially if you need to create package binaries for different configurations, because you would have to export-sources all of them at the same time.

In other words, the exports and exports_sources must be completely invariant for all different configurations and binaries.

If you are packaging pre-compiled binaries, you want to use the conan export-pkg command instead, please have a look to: https://docs.conan.io/2/tutorial/creating_packages/other_types_of_packages/package_prebuilt_binaries.html

[Peddaahh]

Thank you for your inputs, set it up and is running well now