search

conan-io / conan-package-tools

Conan Package Tools. Helps with massive package creation and CI integration (Travis CI, Appveyor...)
MIT License
166 stars 71 forks source link
ci-server conan conan-docker docker-image travis-ci

readme

.github/workflows/conan_package_tools.yml codecov PyPI - Downloads

This project is no longer recommended or maintained 🛑

This project is no longer maintained, it will not get any fixes or support. It will be soon fully archived. Modern Conan 2.0 extensions can be found in https://github.com/conan-io/conan-extensions

Conan 2.0 support :warning:

The project Conan Package Tools does not support Conan 2.x and there is no current planned support.

In case you need such support, please, open an issue explaining your current case with more details.

Conan Package Tools

Introduction

This package allows to automate the creation of conan packages for different configurations.

It eases the integration with CI servers like TravisCI and Appveyor, so you can use the cloud to generate different binary packages for your conan recipe.

Also supports Docker to create packages for different GCC and Clang versions.

Installation

$ pip install conan_package_tools

Or you can clone this repository and store its location in PYTHONPATH.

How it works

Using only conan C/C++ package manager (without conan package tools), you can use the conan create command to generate, for the same recipe, different binary packages for different configurations. The easier way to do it is using profiles:

$ conan create myuser/channel --profile win32
$ conan create myuser/channel --profile raspi
$ ...

The profiles can contain, settings, options, environment variables and build_requires. Take a look to the conan docs to know more.

Conan package tools allows to declare (or autogenerate) a set of different configurations (different profiles). It will call conan create for each one, uploading the generated packages to a remote (if needed), and using optionally docker images to ease the creation of different binaries for different compiler versions (gcc and clang supported).

Basic, but not very practical, example

Create a build.py file in your recipe repository, and add the following lines:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(username="myusername")
    builder.add(settings={"arch": "x86", "build_type": "Debug"},
                options={}, env_vars={}, build_requires={})
    builder.add(settings={"arch": "x86_64", "build_type": "Debug"},
                options={}, env_vars={}, build_requires={})
    builder.run()

Now we can run the python script, the ConanMutiPackager will run the conan create command two times, one to generate x86 Debug package and another one for x86_64 Debug.

> python build.py

############## CONAN PACKAGE TOOLS ######################

INFO: ******** RUNNING BUILD **********
conan create myuser/testing --profile /var/folders/y1/9qybgph50sjg_3sm2_ztlm6dr56zsd/T/tmpz83xXmconan_package_tools_profiles/profile

[build_requires]
[settings]
arch=x86
build_type=Debug
[options]
[scopes]
[env]

...

############## CONAN PACKAGE TOOLS ######################

INFO: ******** RUNNING BUILD **********
conan create myuser/testing --profile /var/folders/y1/9qybgph50sjg_3sm2_ztlm6dr56zsd/T/tmpMiqSZUconan_package_tools_profiles/profile

[build_requires]
[settings]
arch=x86_64
build_type=Debug
[options]
[scopes]
[env]

#########################################################

...

If we inspect the local cache we can see that there are two binaries generated for our recipe, in this case the zlib recipe:

$ conan search zlib/1.2.11@myuser/testing

Existing packages for recipe zlib/1.2.11@myuser/testing:

Package_ID: a792eaa8ec188d30441564f5ba593ed5b0136807
    [options]
        shared: False
    [settings]
        arch: x86
        build_type: Debug
        compiler: apple-clang
        compiler.version: 9.0
        os: Macos
    outdated from recipe: False

Package_ID: e68b263f26a4d7513e28c9cae1673aa0466af777
    [options]
        shared: False
    [settings]
        arch: x86_64
        build_type: Debug
        compiler: apple-clang
        compiler.version: 9.0
        os: Macos
    outdated from recipe: False

Now, we could add new build configurations, but in this case we only want to add Visual Studio configurations and the runtime, but, of course, only if we are on Windows:

import platform
from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(username="myusername")
    if platform.system() == "Windows":
        builder.add(settings={"arch": "x86", "build_type": "Debug", "compiler": "Visual Studio", "compiler.version": 14, "compiler.runtime": "MTd"},
                    options={}, env_vars={}, build_requires={})
        builder.add(settings={"arch": "x86_64", "build_type": "Release", "compiler": "Visual Studio", "compiler.version": 14, "compiler.runtime": "MT"},
                    options={}, env_vars={}, build_requires={})
    else:
        builder.add(settings={"arch": "x86", "build_type": "Debug"},
                    options={}, env_vars={}, build_requires={})
        builder.add(settings={"arch": "x86_64", "build_type": "Debug"},
                    options={}, env_vars={}, build_requires={})
    builder.run()

In the previous example, when we are on Windows, we are adding two build configurations:

- "Visual Studio 14, Debug, MTd runtime"
- "Visual Studio 14, Release, MT runtime"

We can also adjust the options, environment variables and build_requires:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(username="myuser")
    builder.add({"arch": "x86", "build_type": "Release"},
                {"mypackage:option1": "ON"},
                {"PATH": "/path/to/custom"},
                {"*": ["MyBuildPackage/1.0@lasote/testing"]})
    builder.add({"arch": "x86_64", "build_type": "Release"}, {"mypackage:option1": "ON"})
    builder.add({"arch": "x86", "build_type": "Debug"}, {"mypackage:option2": "OFF", "mypackage:shared": True})
    builder.run()

We could continue adding configurations, but probably you realized that it would be such a tedious task if you want to generate many different configurations in different operating systems, using different compilers, different compiler versions etc.

Generating the build configurations automatically

Conan package tools can generate automatically a matrix of build configurations combining architecture, compiler, compiler.version, compiler.runtime, compiler.libcxx, build_type and and shared/static options.

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager()
    builder.add_common_builds()
    builder.run()

If you run the python build.py command, for instance, in Mac OSX, it will add the following configurations automatically:

{'compiler.version': '7.3', 'arch': 'x86', 'build_type': 'Release', 'compiler': 'apple-clang'})
{'compiler.version': '7.3', 'arch': 'x86', 'build_type': 'Debug', 'compiler': 'apple-clang'})
{'compiler.version': '7.3', 'arch': 'x86_64', 'build_type': 'Release', 'compiler': 'apple-clang'})
{'compiler.version': '7.3', 'arch': 'x86_64', 'build_type': 'Debug', 'compiler': 'apple-clang'})
{'compiler.version': '8.0', 'arch': 'x86', 'build_type': 'Release', 'compiler': 'apple-clang'})
{'compiler.version': '8.0', 'arch': 'x86', 'build_type': 'Debug', 'compiler': 'apple-clang'})
{'compiler.version': '8.0', 'arch': 'x86_64', 'build_type': 'Release', 'compiler': 'apple-clang'})
{'compiler.version': '8.0', 'arch': 'x86_64', 'build_type': 'Debug', 'compiler': 'apple-clang'})
{'compiler.version': '8.1', 'arch': 'x86', 'build_type': 'Release', 'compiler': 'apple-clang'})
{'compiler.version': '8.1', 'arch': 'x86', 'build_type': 'Debug', 'compiler': 'apple-clang'})
{'compiler.version': '8.1', 'arch': 'x86_64', 'build_type': 'Release', 'compiler': 'apple-clang'})
{'compiler.version': '8.1', 'arch': 'x86_64', 'build_type': 'Debug', 'compiler': 'apple-clang'})

These are all the combinations of arch=x86/x86_64, build_type=Release/Debug for different compiler versions.

But having different apple-clang compiler versions installed in the same machine is not common at all. We can adjust the compiler versions using a parameter or an environment variable, specially useful for a CI environment:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(apple_clang_versions=["9.0"]) # or declare env var CONAN_APPLE_CLANG_VERSIONS=9.0
    builder.add_common_builds()
    builder.run()

In this case, it will call conan create with only this configurations:

{'compiler.version': '9.0', 'arch': 'x86', 'build_type': 'Release', 'compiler': 'apple-clang'})
{'compiler.version': '9.0', 'arch': 'x86', 'build_type': 'Debug', 'compiler': 'apple-clang'})
{'compiler.version': '9.0', 'arch': 'x86_64', 'build_type': 'Release', 'compiler': 'apple-clang'})
{'compiler.version': '9.0', 'arch': 'x86_64', 'build_type': 'Debug', 'compiler': 'apple-clang'})

You can adjust other constructor parameters to control the build configurations that will be generated:

Or you can adjust environment variables:

Check the REFERENCE section to see all the parameters and ENVIRONMENT VARIABLES available.

IMPORTANT! Both the constructor parameters and the corresponding environment variables of the previous list ONLY have effect when using builder.add_common_builds().

So, if we want to generate packages for x86_64 and armv8 but only for Debug and apple-clang 9.0:

$ export CONAN_ARCHS=x86_64,armv8
$ export CONAN_APPLE_CLANG_VERSIONS=9.0
$ export CONAN_BUILD_TYPES=Debug

$ python build.py

There are also two additional parameters of the add_common_builds:

    def configure(self):
        del self.settings.compiler.libcxx
from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager()
    builder.add_common_builds(shared_option_name="mypackagename:shared", pure_c=False)
    builder.run()

Filtering or modifying the configurations

Use the remove_build_if helper with a lambda function to filter configurations:

from cpt.packager import ConanMultiPackager

builder = ConanMultiPackager(username="myuser")
builder.add_common_builds()
builder.remove_build_if(lambda build: build.settings["compiler.version"] == "4.6" and build.settings["build_type"] == "Debug")

Use the update_build_if helper with a lambda function to alter configurations:

from cpt.packager import ConanMultiPackager

builder = ConanMultiPackager(username="myuser")
builder.add_common_builds()
builder.update_build_if(lambda build: build.settings["os"] == "Windows",
                        new_build_requires={"*": ["7zip_installer/0.1.0@conan/stable"]})
# Also avaiable parameters:
#    new_settings, new_options, new_env_vars, new_build_requires, new_reference

Or you can directly iterate the builds to do any change. EX: Remove the GCC 4.6 packages with build_type=Debug:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(username="myuser")
    builder.add_common_builds()
    filtered_builds = []
    for settings, options, env_vars, build_requires, reference in builder.items:
        if settings["compiler.version"] != "4.6" and settings["build_type"] != "Debug":
             filtered_builds.append([settings, options, env_vars, build_requires, reference])
    builder.builds = filtered_builds
    builder.run()

Package Version based on Commit Checksum

Sometimes you want to use Conan as in-source but you do not need to specify a version in the recipe, it could be configured by your build environment. Usually you could use the branch name as the package version, but if you want to create unique packages for each new build, upload it and do not override on your remote, you will need to use a new version for each build. In this case, the branch name will not be enough, so a possible approach is to use your current commit checksum as version:

from cpt.packager import ConanMultiPackager
from cpt.ci_manager import CIManager
from cpt.printer import Printer

if __name__ == "__main__":
    printer = Printer()
    ci_manager = CIManager(printer)
    builder = ConanMultiPackager(reference="mypackage/{}".format(ci_manager.get_commit_id()[:7]))
    builder.add_common_builds()
    builder.run()

As SHA-1 is 40 digits long, you could format the result to short size

Save created packages summary

In case you want to integrate CPT with other tools, for example you want to have build logic after creating packages, you can save a json report about all configurations and packages.

Examples:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager()
    builder.add_common_builds()
    builder.run(summary_file='cpt_summary_file.json')

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager()
    builder.add_common_builds()
    builder.run()
    builder.save_packages_summary('cpt_summary_file.json')

Alternatively you can use the CPT_SUMMARY_FILE environment variable to set the summary file path

Using all values for custom options

Sometimes you want to include more options to your matrix, including all possible combinations, so that, you can use build_all_options_values:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(reference="mypackage/0.1.0")
    builder.add_common_builds(build_all_options_values=["mypackage:foo", "mypackage:bar"])
    builder.run()

Now let's say mypackage's recipe contains the follow options: shared, fPIC, foo and bar. Both foo and bar can accept True or False. The method add_common_builds will generate a matrix including both foo and bar with all possible combinations.

Using Docker

Instance ConanMultiPackager with the parameter use_docker=True, or declare the environment variable CONAN_USE_DOCKER: It will launch, when needed, a container for the current build configuration that is being built (only for Linux builds).

There are docker images available for different gcc versions: 4.6, 4.8, 4.9, 5, 6, 7 and clang versions: 3.8, 3.9, 4.0.

The containers will share the conan storage directory, so the packages will be generated in your conan directory.

Example:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager()
    builder.add_common_builds()
    builder.run()

And run the build.py:

$ export CONAN_USERNAME=myuser
$ export CONAN_GCC_VERSIONS=4.9
$ export CONAN_DOCKER_IMAGE=conanio/gcc49
$ export CONAN_USE_DOCKER=1
$ python build.py

It will generate a set of build configurations (profiles) for gcc 4.9 and will run it inside a container of the conanio/gcc49 image.

If you want to run the arch="x86" build inside a docker container of 32 bits you can set the parameter docker_32_images in the ConanMultiPackager constructor or set the environment variable CONAN_DOCKER_32_IMAGES. In this case, the docker image name to use will be appended with -i386.

The Docker images used by default both for 64 and 32 bits are pushed to dockerhub and its Dockerfiles are available in the conan-docker-tools repository.

Running scripts and executing commands before to build on Docker

When Conan Package Tools uses Docker to build your packages, sometimes you need to execute a "before build" step. If you need to install packages, change files or create a setup, there is an option for that: docker_entry_script

Example:

This example shows how to install tzdata package by apt-get, before to build the Conan package.

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    command = "sudo apt-get -qq update && sudo apt-get -qq install -y tzdata"
    builder = ConanMultiPackager(use_docker=True, docker_image='conanio/gcc7', docker_entry_script=command)
    builder.add_common_builds()
    builder.run()

Also, it's possible to run some internal script, before to build the package:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    command = "python bootstrap.py"
    builder = ConanMultiPackager(use_docker=True, docker_image='conanio/gcc7', docker_entry_script=command)
    builder.add_common_builds()
    builder.run()

Using with your own Docker images

The default location inside the Docker container is /home/conan on Linux and C:\Users\ContainerAdministrator on Windows. This is fine if you use the conan Docker images but if you are using your own image, these locations probably won't exist.

To use a different location, you can use the option docker_conan_home or the environment variable CONAN_DOCKER_HOME.

Installing extra python packages before to build

Maybe you need to install some python packages using pip before to build your conan package. To solve this situation you could use pip_install:

Example:

This example installs bincrafters-package-tools and conan-promote before to build:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(pip_install=["bincrafters-package-tools==0.17.0", "conan-promote==0.1.2"])
    builder.add_common_builds()
    builder.run()

But if you prefer to use environment variables:

export CONAN_PIP_INSTALL="bincrafters-package-tools==0.17.0,conan-promote=0.1.2"

Passing additional Docker parameters during build

When running conan create step in Docker, you might want to run the container with a different Docker network. For this you can use docker_run_options parameter (or CONAN_DOCKER_RUN_OPTIONS envvar)

builder = ConanMultiPackager(
  docker_run_options='--network bridge --privileged',
  ...

When run, this will translate to something like this:

sudo -E docker run ... --network bridge --privileged conanio/gcc6 /bin/sh -c "cd project &&  run_create_in_docker"

Installing custom Conan config

To solve custom profiles and remotes, Conan provides the config feature where is possible to edit the conan.conf or install config files.

If you need to run conan config install <url> before to build there is the argument config_url in CPT:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    config_url = "https://github.com/bincrafters/conan-config.git"
    builder = ConanMultiPackager(config_url=config_url)
    builder.add_common_builds()
    builder.run()

But if you are not interested to update your build.py script, it's possible to use environment variables instead:

export CONAN_CONFIG_URL=https://github.com/bincrafters/conan-config.git

Specifying a different base profile

The options, settings and environment variables that the add_common_builds() method generate, are applied into the default profile of the conan installation. If you want to use a different profile you can pass the name of the profile in the run() method.

Example:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(clang_versions=["3.8", "3.9"])
    builder.add_common_builds()
    builder.run("myclang")

Alternatively you can use the CONAN_BASE_PROFILE environment variable to choose a different base profile:

CONAN_BASE_PROFILE=myclang

Specifying build context for cross building

Since Conan 1.24, you can pass an additional profile for build context, so that, you can pass both profiles by environment variables:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(gcc_versions=["10", "11"])
    builder.add_common_builds()
    builder.run(base_profile_name="raspberrypi", base_profile_build_name="default")

The base_profile_name is equivalent to profile host, where my libraries and executables will run, and the base_profile_build_name is the profile related where the artifacts are built.

Also, you can use environment variables instead:

CONAN_BASE_PROFILE=default
CONAN_BASE_PROFILE_BUILD=raspberrypi

Read more build context here

The CI integration

If you are going to use a CI server to generate different binary packages for your recipe, the best approach is to control the build configurations with environment variables.

So, having a generic build.py should be enough for almost all the cases:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager()
    builder.add_common_builds(shared_option_name="mypackagename:shared", pure_c=False)
    builder.run()

Then, in your CI configuration, you can declare different environment variables to limit the build configurations to an specific compiler version, using a specific docker image etc.

For example, if you declare the following environment variables:

CONAN_GCC_VERSIONS=4.9
CONAN_DOCKER_IMAGE=conanio/gcc49

the add_common_builds() method will only add different build configurations for GCC=4.9 and will run them in a docker container.

You can see working integrations with Travis and Appveyor in the zlib repository here

Travis integration

Travis CI can generate a build with multiple jobs defining a matrix with environment variables. We can configure the builds to be executed in the jobs by defining some environment variables.

The following is a real example of a .travis.yml file that will generate packages for Linux gcc (4.9, 5, 6), Linux Clang (3.9 and 4.0) and OSx with apple-clang (8.0, 8.1 and 9.0).

Remember, you can use conan new command to generate the base files for appveyor, travis etc. Check conan new --help.

.travis.yml example:

env:
   global:
     - CONAN_USERNAME: "conan" # ADJUST WITH YOUR REFERENCE USERNAME!
     - CONAN_LOGIN_USERNAME: "lasote" # ADJUST WITH YOUR LOGIN USERNAME!
     - CONAN_CHANNEL: "testing" # ADJUST WITH YOUR CHANNEL!
     - CONAN_UPLOAD: "https://api.bintray.com/conan/conan-community/conan" # ADJUST WITH YOUR REMOTE!
     - CONAN_STABLE_BRANCH_PATTERN: "release/*"
     - CONAN_UPLOAD_ONLY_WHEN_STABLE: 1 # Will only upload when the branch matches "release/*"

linux: &linux
   os: linux
   sudo: required
   language: python
   python: "3.6"
   services:
     - docker
osx: &osx
   os: osx
   language: generic
matrix:
   include:

      - <<: *linux
        env: CONAN_GCC_VERSIONS=4.9 CONAN_DOCKER_IMAGE=conanio/gcc49
      - <<: *linux
        env: CONAN_GCC_VERSIONS=5 CONAN_DOCKER_IMAGE=conanio/gcc5
      - <<: *linux
        env: CONAN_GCC_VERSIONS=6 CONAN_DOCKER_IMAGE=conanio/gcc6
      - <<: *linux
        env: CONAN_GCC_VERSIONS=7 CONAN_DOCKER_IMAGE=conanio/gcc7
      - <<: *linux
        env: CONAN_CLANG_VERSIONS=3.9 CONAN_DOCKER_IMAGE=conanio/clang39
      - <<: *linux
        env: CONAN_CLANG_VERSIONS=4.0 CONAN_DOCKER_IMAGE=conanio/clang40
      - <<: *osx
        osx_image: xcode7.3
        env: CONAN_APPLE_CLANG_VERSIONS=7.3
      - <<: *osx
        osx_image: xcode8.2
        env: CONAN_APPLE_CLANG_VERSIONS=8.0
      - <<: *osx
        osx_image: xcode8.3
        env: CONAN_APPLE_CLANG_VERSIONS=8.1
      - <<: *osx
        osx_image: xcode9
        env: CONAN_APPLE_CLANG_VERSIONS=9.0

install:
  - chmod +x .travis/install.sh
  - ./.travis/install.sh

script:
  - chmod +x .travis/run.sh
  - ./.travis/run.sh

You can also use multiples "pages" to split the builds in different jobs (Check pagination section first to understand):

.travis.yml

env:
   global:
     - CONAN_USERNAME: "conan" # ADJUST WITH YOUR REFERENCE USERNAME!
     - CONAN_LOGIN_USERNAME: "lasote" # ADJUST WITH YOUR LOGIN USERNAME!
     - CONAN_CHANNEL: "testing" # ADJUST WITH YOUR CHANNEL!
     - CONAN_UPLOAD: "https://api.bintray.com/conan/conan-community/conan" # ADJUST WITH YOUR REMOTE!
     - CONAN_STABLE_BRANCH_PATTERN: "release/*"
     - CONAN_UPLOAD_ONLY_WHEN_STABLE: 1 # Will only upload when the branch matches "release/*"

linux: &linux
   os: linux
   sudo: required
   language: python
   python: "3.6"
   services:
     - docker
osx: &osx
   os: osx
   language: generic
matrix:
   include:

      - <<: *linux
        env: CONAN_GCC_VERSIONS=4.9 CONAN_DOCKER_IMAGE=conanio/gcc49 CONAN_CURRENT_PAGE=1

      - <<: *linux
        env: CONAN_GCC_VERSIONS=4.9 CONAN_DOCKER_IMAGE=conanio/gcc49 CONAN_CURRENT_PAGE=2

      - <<: *linux
        env: CONAN_GCC_VERSIONS=5 CONAN_DOCKER_IMAGE=conanio/gcc5 CONAN_CURRENT_PAGE=1

       - <<: *linux
        env: CONAN_GCC_VERSIONS=5 CONAN_DOCKER_IMAGE=conanio/gcc5 CONAN_CURRENT_PAGE=2

      - <<: *linux
        env: CONAN_GCC_VERSIONS=6 CONAN_DOCKER_IMAGE=conanio/gcc6 CONAN_CURRENT_PAGE=1

      - <<: *linux
        env: CONAN_GCC_VERSIONS=6 CONAN_DOCKER_IMAGE=conanio/gcc6 CONAN_CURRENT_PAGE=2

      - <<: *linux
        env: CONAN_CLANG_VERSIONS=3.9 CONAN_DOCKER_IMAGE=conanio/clang39 CONAN_CURRENT_PAGE=1

       - <<: *linux
        env: CONAN_CLANG_VERSIONS=3.9 CONAN_DOCKER_IMAGE=conanio/clang39 CONAN_CURRENT_PAGE=2

      - <<: *linux
        env: CONAN_CLANG_VERSIONS=4.0 CONAN_DOCKER_IMAGE=conanio/clang40 CONAN_CURRENT_PAGE=1

      - <<: *linux
        env: CONAN_CLANG_VERSIONS=4.0 CONAN_DOCKER_IMAGE=conanio/clang40 CONAN_CURRENT_PAGE=2

      - <<: *osx
        osx_image: xcode7.3
        env: CONAN_APPLE_CLANG_VERSIONS=7.3 CONAN_CURRENT_PAGE=1

      - <<: *osx
        osx_image: xcode7.3
        env: CONAN_APPLE_CLANG_VERSIONS=7.3 CONAN_CURRENT_PAGE=2

      - <<: *osx
        osx_image: xcode8.2
        env: CONAN_APPLE_CLANG_VERSIONS=8.0 CONAN_CURRENT_PAGE=1

      - <<: *osx
        osx_image: xcode8.2
        env: CONAN_APPLE_CLANG_VERSIONS=8.0 CONAN_CURRENT_PAGE=2

      - <<: *osx
        osx_image: xcode8.3
        env: CONAN_APPLE_CLANG_VERSIONS=8.1 CONAN_CURRENT_PAGE=1

      - <<: *osx
        osx_image: xcode8.3
        env: CONAN_APPLE_CLANG_VERSIONS=8.1 CONAN_CURRENT_PAGE=2

install:
  - chmod +x .travis/install.sh
  - ./.travis/install.sh

script:
  - chmod +x .travis/run.sh
  - ./.travis/run.sh

.travis/install.sh

#!/bin/bash

set -e
set -x

if [[ "$(uname -s)" == 'Darwin' ]]; then
    brew update || brew update
    brew outdated pyenv || brew upgrade pyenv
    brew install pyenv-virtualenv
    brew install cmake || true

    if which pyenv > /dev/null; then
        eval "$(pyenv init -)"
    fi

    pyenv install 3.7.11
    pyenv virtualenv 3.7.11 conan
    pyenv rehash
    pyenv activate conan
fi

pip install conan --upgrade
pip install conan_package_tools
conan user

If you want to "pin" a conan_package_tools version use:

pip install conan_package_tools==0.37.0

That version will be used also in the docker images.

.travis/run.sh

#!/bin/bash

set -e
set -x

if [[ "$(uname -s)" == 'Darwin' ]]; then
    if which pyenv > /dev/null; then
        eval "$(pyenv init -)"
    fi
    pyenv activate conan
fi

python build.py

Remember to set the CONAN_PASSWORD variable in the travis build control panel!

Appveyor integration

This is very similar to Travis CI. With the same build.py script we have the following appveyor.yml file:

build: false

environment:
    PYTHON: "C:\\Python37"
    PYTHON_VERSION: "3.7.9"
    PYTHON_ARCH: "32"

    CONAN_USERNAME: "lasote"
    CONAN_LOGIN_USERNAME: "lasote"
    CONAN_CHANNEL: "stable"
    VS150COMNTOOLS: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\Common7\\Tools\\"
    CONAN_UPLOAD: "https://api.bintray.com/conan/luisconanorg/fakeconancenter"
    CONAN_REMOTES: "https://api.bintray.com/conan/luisconanorg/conan-testing"

    matrix:
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
          CONAN_VISUAL_VERSIONS: 12
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
          CONAN_VISUAL_VERSIONS: 14
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017
          CONAN_VISUAL_VERSIONS: 15

install:
  - set PATH=%PATH%;%PYTHON%/Scripts/
  - pip.exe install conan_package_tools --upgrade
  - conan user # It creates the conan data directory

test_script:
  - python build.py

Bamboo CI integration

Bamboo is a commercial CI tool developed by Atlassian. When building from bamboo, several environment variables get set during builds.

If the env var bamboo_buildNumber is set and the branch name (bamboo_planRepository_branch env var) matches stable_branch_pattern, then the channel name gets set to stable.

Jenkins CI integration

Jenkins is an open source CI tool that was originally forked from hudson. When building on jenkins, several environment variables get set during builds.

If the env var JENKINS_URL is set and the branch name (BRANCH_NAME env var) matches stable_branch_pattern, then the channel name gets set to stable.

Currently, only the pipeline builds set the BRANCH_NAME env var automatically.

GitLab CI integration

GitLab CI is a commercial CI tool developed by GitLab. When building on gitlab-ci, several environment variables get set during builds.

If the env var GITLAB_CI is set and the branch name (CI_BUILD_REF_NAME env var) matches stable_branch_pattern, then the channel name gets set to stable.

Upload packages

You can upload the generated packages automatically to a conan-server using the following environment variables (parameters also available):

Upload dependencies (#237)

Sometimes your dependencies are not available in remotes and you need to pass --build=missing to build them. The problem is that you will need to fix one-by-one, updating the CI script, instead of just uploading all built packages.

Now you can upload ALL of your dependencies together, in addition to your package, to the same remote. To do this, you need to define:

CONAN_UPLOAD_DEPENDENCIES="all"

Or, set it in ConanMultiPackager arguments:

ConanMultiPackager(upload_dependencies="all")

However, maybe you want to upload ONLY specified packages by their names:

CONAN_UPLOAD_DEPENDENCIES="foo/0.1@user/channel,bar/1.2@bar/channel"

Or,

ConanMultiPackager(upload_dependencies=["foo/0.1@user/channel", "bar/1.2@bar/channel"])

Pagination

Sometimes, if your library is big or complex enough in terms of compilation time, the CI server could reach the maximum time of execution, because it's building, for example, 20 different binary packages for your library in the same machine.

You can split the different build configurations in different "pages". So, you can configure your CI to run more "worker" machines, one per "page".

There are two approaches:

Sequential distribution

By simply passing two pagination parameters, curpage and total_pages or the corresponding environment variables:

$ export CONAN_TOTAL_PAGES=3
$ export CONAN_CURRENT_PAGE=1

$ python build.py

If you added 10 different build configurations to the builder:

In your CI server you can configure a matrix with different "virtual machines" or "jobs" or "workers": In each "machine" you can specify a different CONAN_CURRENT_PAGE environment variable. So your different configurations will be distributed in the different machines.

Named pages

By adding builds to the named_builds dictionary, and passing curpage with the page name:

from cpt.packager import ConanMultiPackager
from collections import defaultdict

if __name__ == '__main__':
    builder = ConanMultiPackager(curpage="x86", total_pages=2)
    named_builds = defaultdict(list)
    builder.add_common_builds(shared_option_name="bzip2:shared", pure_c=True)
    for settings, options, env_vars, build_requires, reference in builder.items:
        named_builds[settings['arch']].append([settings, options, env_vars, build_requires, reference])
    builder.named_builds = named_builds
    builder.run()

named_builds now have a dictionary entry for x86 and another for x86_64:

Generating multiple references for the same recipe

You can add a different reference in the builds tuple, so for example, if your recipe has no "version" field, you could generate several versions in the same build script. Conan package tools will export the recipe using the different reference automatically:

from cpt.packager import ConanMultiPackager

if __name__ == '__main__':
    builder = ConanMultiPackager()
    builder.add_common_builds(reference="mylib/1.0@conan/stable")
    builder.add_common_builds(reference="mylib/2.0@conan/stable")
    builder.run()

Working with Bintray: Configuring repositories

Use the argument upload or environment variable CONAN_UPLOAD to set the URL of the repository where you want to upload your packages. Will be also used to read from it.

Use CONAN_PASSWORD environment variable to set the API key from Bintray. If your username in Bintray doesn't match with the specified CONAN_USERNAME specify the variable CONAN_LOGIN_USERNAME or the parameter login_username to ConanMultiPackager .

If you are using travis or appveyor you can use a hidden enviroment variable from the repository setup package.

To get an API key in Bintray to "Edit profile"/"API key".

Use the argument remotes or environment variable CONAN_REMOTES to configure additional repositories containing needed requirements.

Example: Add your personal Bintray repository to retrieve and upload your packages and also some other different repositories to read some requirements.

In your .travis.yml or appveyor.yml files declare the environment variables:

CONAN_UPLOAD="https://api.bintray.com/mybintrayuser/conan_repository"
CONAN_REMOTES="https://api.bintray.com/other_bintray_user/conan-repo, https://api.bintray.com/other_bintray_user2/conan-repo"

Or in your build.py:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(username="myuser",
                                 upload="https://api.bintray.com/mybintrayuser/conan_repository",
                                 remotes="https://api.bintray.com/other_bintray_user/conan-repo, https://api.bintray.com/other_bintray_user2/conan-repo")
    builder.add_common_builds()
    builder.run()

Visual Studio auto-configuration

When the builder detects a Visual Studio compiler and its version, it will automatically configure the execution environment for the "conan test" command with the vcvarsall.bat script (provided by all Microsoft Visual Studio versions). So you can compile your project with the right compiler automatically, even without CMake.

MinGW builds

MinGW compiler builds are also supported. You can use this feature with Appveyor.

You can choose different MinGW compiler configurations:

Using MINGW_CONFIGURATIONS env variable in Appveyor:

MINGW_CONFIGURATIONS: '4.9@x86_64@seh@posix, 4.9@x86_64@seh@win32'

Check an example here

Clang builds

Clang compiler builds are also supported. You can use this feature with TravisCI.

You can choose different Clang compiler configurations:

Using CONAN_CLANG_VERSIONS env variable in Travis ci or Appveyor:

CONAN_CLANG_VERSIONS = "3.8,3.9,4.0"

FULL REFERENCE

ConanMultiPackager parameters reference

Upload related parameters:

Commit messages reference

The current commit message can contain special messages:

Complete ConanMultiPackager methods reference:

Environment configuration

You can also use environment variables to change the behavior of ConanMultiPackager, so that you don't pass parameters to the ConanMultiPackager constructor.

This is especially useful for CI integration.

Full example

You can see the full zlib example here