Avoid unnecessary build/package in CI

[boussaffawalid]

Hello,

I'm trying to optimize our CI build. The idea is to avoid building a package if it already exist in remote. (same git hash, same options ,same settings ...)

One way to achieve this is to use package id and check if it already exist.

I found this method that provide the package id:

self.info.package_id()

However, it is only accessible in the package method.

Is there a way a way to compute the package_id before creating it ?

Thanks, Walid

[Johnnyxy]

Checkout the Conan-Package-Tools (CPT). It exactly does this automatically already (see here).

On our CI system this behaviour is a problem as there a package should be built regardless if it already exists (but not the dependencies) (see here). One can force the CPT to instruct Conan to build (e.g. env var CONAN_BUILD_POLICY=always) but this would also force to build dependencies not only the specific package.

[boussaffawalid]

Thank you Johnnyxy! Actually we already use Conan-Package-Tools (CPT) but I I was not aware of this option. I tested it already and it works like a charm 🥇

The only issue now is that the information is not accessible outside CPT. (is it ?) We have different CI logic depends on if the package was built of not.

Is it possible for example to save some kind of report with CPT that can be consumed later on ?

[Johnnyxy]

What do you mean with report? What information do you seek?

Do you mean that it should report if something has been built or not? At the moment I am not aware of any approach that gives you this information.

You could check if a package already exists by using conan search (see here) with its query parameter -q to check if a package with given settings/options exists.

What else you could do is to write something to the filesystem in the def build(self) function in your recipe. Like writing a file I_have_been_built.txt and check for that later on. I am not using this approach but at CMake time CPack creates a ZIP file of the built files (all executed def build(self)). This file only exists is CPack ran. And CPack only runs if something had been built thus an existing ZIP file indicates a built package.

[boussaffawalid]

I mean like a json file for example contains some info about what CPT actually did.

• what was built
• what was uploaded
• options/settings ... ...

Currently I write something to disk in the def build(self)method, I just think it is hacky and It would be great if CPT could report back in a machine readable format (json/yaml ...) rather than just printing logs.

[Johnnyxy]

I am not aware of any reporting of CPT. For that you should file a feature-request at CPT's issue tracker. But as I saw you already did something related there :) https://github.com/conan-io/conan-package-tools/issues/392

[jgsogo]

Hi! First of all a warning ⚠️ : CPT is a library we are using to help with massive binary generation on our systems. It is not recommended for public usage as it can be broken at any time or even deprecated.

@boussaffawalid , you can get the package ID without creating it using conan info <conanfile.py> [--profile]. And also you can have a look at the lockfile feature, it will give you much more information than the package ID: here you have what was built (nodes labeled with modified: true), what was downloaded (all the dependencies), options/settings,... I paste here an example:

{  
   "profile":"[settings]\narch=x86_64\narch_build=x86_64\nbuild_type=Release\ncompiler=gcc\ncompiler.libcxx=libstdc++\ncompiler.version=7\nos=Linux\nos_build=Linux\n[options]\n[build_requires]\n[env]\n",
   "graph_lock":{  
      "nodes":{  
         "f0f73f38-c98b-11e9-93ab-0242ac130007":{  
            "pref":"protoc_installer/3.6.1@bincrafters/stable#0:25c81eab3877cbaef5f20ea104197e80631b7664#cd56cbdf0ce17a1fa4fbc2b260b48f0a",
            "options":"protobuf:fPIC=True\nprotobuf:lite=False\nprotobuf:shared=False\nprotobuf:with_zlib=False",
            "requires":{  
               "protobuf/3.6.1@bincrafters/stable#0":"f0f73c2c-c98b-11e9-93ab-0242ac130007"
            }
         },
         "f0f74118-c98b-11e9-93ab-0242ac130007":{  
            "pref":"protobuf/3.6.1@bincrafters/stable#0:d351525cc53ebe68279edf1978846402420066e7#0",
            "options":"fPIC=True\nlite=False\nshared=False\nwith_zlib=False"
         },
         "f0f742f8-c98b-11e9-93ab-0242ac130007":{  
            "pref":"core-messages/0.0@sword/sorcery#334d2a661086b8676022a1e35eea12a5:bbeef259ed68d0bfcdfe6c6e7c5f8d3581876931#ea71e98f620240abc56d5b148015d9f0",
            "options":"shared=False\nprotobuf:fPIC=True\nprotobuf:lite=False\nprotobuf:shared=False\nprotobuf:with_zlib=False",
            "modified":true,
            "requires":{  
               "protobuf/3.6.1@bincrafters/stable#0":"f0f74118-c98b-11e9-93ab-0242ac130007",
               "protoc_installer/3.6.1@bincrafters/stable":"f0f73f38-c98b-11e9-93ab-0242ac130007"
            }
         },
         "f0f73c2c-c98b-11e9-93ab-0242ac130007":{  
            "pref":"protobuf/3.6.1@bincrafters/stable#0:d351525cc53ebe68279edf1978846402420066e7#0",
            "options":"fPIC=True\nlite=False\nshared=False\nwith_zlib=False"
         }
      }
   }
}

---

Anyway, I would really like to know the workflow you are using for you CI (I'm gathering scenarios here https://github.com/conan-io/conan/issues/5553), the more scenarios we know about, the more general our solution will be.

[AbrilRBS]

Closing as solved in Conan 2. conan graph subcommands already output the necessary information for efficient CI rebuilding of only necessary packages (conan graph info for a specific graph would give its status, and conan graph build-order would output the proper order of the necessary references)