Integrate oo-cmake with biicode C/C++ dependency management

[Manu343726]

Hi! I'm Manu Sánchez, from the biicode team.

I'm working intensively with CMake scripts and I find your library so useful to me. Maps, function calls, etc; features that make feasible writing complex CMake scripts without going crazy!

biicode is a file-based dependency manager for C and C++, in the same spirit of Java's Maven and Maven Central or Python's pip. The tool uses CMake under the hood to process builds, so to setup a biicode block* you should play a bit with CMakeLists.txt files. A block is our unit of code sharing, think of it as a package.
Here's an example, part of the CMakeLists.txt file of our Box2D block:

IF(BIICODE)
    INCLUDE(${CMAKE_HOME_DIRECTORY}/biicode.cmake)
    INIT_BIICODE_BLOCK()
    ADD_DEFINITIONS(-DGLEW_STATIC)
    SET(BII_LIB_SYSTEM_DEPS )
    ADD_BIICODE_TARGETS()
    target_include_directories(${BII_LIB_TARGET} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
ELSE()
    cmake_minimum_required(VERSION 2.6)

    project(Box2D)

    if(UNIX)
        set(BOX2D_INSTALL_BY_DEFAULT ON)
    else(UNIX)
        set(BOX2D_INSTALL_BY_DEFAULT OFF)
    endif(UNIX)

    option(BOX2D_INSTALL "Install Box2D libs, includes, and CMake scripts" ${BOX2D_INSTALL_BY_DEFAULT})
    option(BOX2D_INSTALL_DOC "Install Box2D documentation" OFF)
    ...
ENDIF()

It's exactly the same CMakeLists.txt from the original library, just adding a couple of toggles to make it work with biicode.
To use Box2D, you only have to #include it in your C/C++ code, biicode does everything else (Download, setup build, etc). Check our docs for more info.

Since version 2.0 we support dependency management of CMake scripts, via include() directives. Your library was very useful to me, so I created a block to deploy and reuse it easily across all my biicode blocks. Now I'm able to use your library inside my biicode CMakeLists.txt files just doing include(manu343726/oo-cmake/oo-cmake).

I hope you like it.

oo-cmake changes

I tried not to touch your codebase at all, but use Travis CI to build and publish the block from your sources. So the only changes I propose you should be on the .travis.yml file (Among other minor changes on the README).

The block

The idea is to have a biicode block with all your sources, while users access functionality via include(manu343726/oo-cmake/oo-cmake). Check the block here.

Dependency management

biicode is a file-based dependency management system. That means biicode uses files as its unit of dependency. So if you use foo.cpp from a manu343726/lib block, biicode will download the foo.cpp file only, not the entire manu343726/lib block. This is interesting for large libraries and codebases, because only the #included files will be downloaded.

But that does not work in this case. Your oo-cmake.cmake file depends in many CMake scripts and some extra C++ files that should be retrieved too, but you are not using our include(user/block/script) convention (And I'm not going to change your sources), so this dependency retrieving does not work directly.

However, biicode allows you to specify dependencies manually in the biicode.conf file, the configuration file of the block:

[dependencies]
    # Manual adjust file implicit dependencies, add (+), remove (-), or overwrite (=)
    # hello.h + hello_imp.cpp hello_imp2.cpp
    # *.h + *.cpp
    oo-cmake.cmake + build/* cmake/* resources/* src/*

In our case, I specified explicitly that oo-cmake.cmake depends on the contents of the build/, cmake/, resources/, and src/ folders. Now biicode will download that folders when the oo-cmake.cmake file is used.

CMakeLists.txt

Each biicode block has it's own CMakeLists.txt with building configuration. If no CMakeLists.txt is provided, biicode generates one. By default biicode searches for any C/C++ file adding it to block targets (Library or executable, depending on the kind of file: Is it a header?, is it a .cpp with a main() function?, etc).

This process, done by the ADD_BIICODE_TARGETS() macro, works like a charm with C/C++ libraries, but not with CMake libraries that use some C/C++ files like yours. We don't want any C/C++ target in the oo-cmake block, we don't even want a target at all since this is designed to be include()d only.

The generated target is called manu343726_oo-cmake_src_cmakeJsonParser_cmakeJsonParser. SInce we don't want that target to be builded as part of our block, disable it after the ADD_BIICODE_TARGETS() call:

INIT_BIICODE_BLOCK()
ADD_BIICODE_TARGETS()

set_target_properties(manu343726_oo-cmake_src_cmakeJsonParser_cmakeJsonParser                                                      
                          PROPERTIES EXCLUDE_FROM_ALL 1 EXCLUDE_FROM_DEFAULT_BUILD 1) 

Generating and publishing the block automatically

So we have to get the block, copy your library sources on it, write the custom biicode.conf, and edit the default biicode CMakeLists.txt to disable the undesired target. Then publish the block.

If you suppose the block already exists this is easier, we just need to open the block and update its contents:

$ bii init #Initialize biicode project
$ bii open manu343726/oo-cmake #Download the entire block for editing
$ cp YOUR_SOURCES ./blocks/manu343726/oo-cmake/
$ bii user manu343726 #We need to log in to publish a block
$ bii publish manu343726/oo-cmake

What I did is to create the block using the web inteface, add the custom biicode.conf and CMakeLists.txt to it, and working in travis starting with that:

install: 
- wget http://apt.biicode.com/install.sh && chmod +x install.sh && ./install.sh
- bii setup:cpp
script:
- bii init
- bii open manu343726/oo-cmake
- rsync -av --exclude="blocks" --exclude="bii" --exclude=".git" --exclude=".gitignore" . blocks/manu343726/oo-cmake
- cmake -P build/script.cmake
after_success: cmake -P build/after_success.cmake
after_failure: cmake -P build/after_failure.cmake
after_script: cmake -P build/after_script.cmake
deploy:
  provider: biicode
  user: manu343726
  password:
    secure: MY ENCRYPTED BIICODE PASSWORD
  skip_cleanup: true #The biicode block is generated during build, don't discard build changes!

Note that Travis CI supports biicode deployment out of the box, just install the travis gem and run biicode setup inside your project:

$ gem install travis
$ cd oo-cmake
$ travis setup biicode

This command will ask you for biicode credentials, some configuration flags, and then will add the deploy entry to your travis.yml.

Accepting changes

This changes cannot be accepted directly via a pull request, because some changes depend on configuration based on my project:

• The biicode block is published and owned by my account: All the examples described above use my manu343726 account, and the block is manu343726/oo-cmake. If you accept this proposal I suppose that you want your own biicode account and to release the oo-cmake with it. That said, I have no problem with owning the block, but this is not possible if you merge the proposal into your project (See next point).
• The travis encryption is based on my project. Travis encryption does not work across forks. So even if you like to deploy the library to biicode using my account, the deploy will not work since you need my password to publish the block.

As I said previously, I tried not to modify your sources. Actually, this is the diff between my fork and your master branch:

@@ -3,24 +3,22 @@ os:
  - linux
  - osx
 compiler:
-  - gcc
+ - gcc
 matrix:
   allow_failures:
-    - os: osx
+   - os: osx
 #env:
 # - CMAKE_DIR="v3.0" CMAKE_VERSION="3.0.1"
 # - CMAKE_DIR="v2.8" CMAKE_VERSION="2.8.12.2"
-notifications:
-  slack: fallto:MlnVOMNkx8YopsaSSxqh2Rcr
 before_install:
-  - sudo apt-get install cmake
-  - sudo apt-get install mercurial
-  - sudo apt-get install git  
-  - sudo apt-get install subversion
-  - git config --global user.email "travis-ci@example.com"
-  - git config --global user.name "Build Server"
-install:
-     # install cmake --
+ - sudo apt-get install cmake
+ - sudo apt-get install mercurial
+ - sudo apt-get install git
+ - sudo apt-get install subversion
+ - git config --global user.email "travis-ci@example.com"
+ - git config --global user.name "Build Server"
+install: 
+    # install cmake --
     #- wget "http://www.cmake.org/files/${CMAKE_DIR}/cmake-${CMAKE_VERSION}.tar.gz"
     #- tar xf "cmake-${CMAKE_VERSION}.tar.gz"
     #- cmake -Hcmake-${CMAKE_VERSION} -B_builds -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="`pwd`/_install"
 @@ -31,12 +29,26 @@ install:
     #- which cmake
     #- cmake --version    
     # -- end
-script: "cmake -P build/script.cmake"
-after_success: "cmake -P build/after_success.cmake"
-after_failure: "cmake -P build/after_failure.cmake"
-after_script: "cmake -P build/after_script.cmake"
+ - wget http://apt.biicode.com/install.sh && chmod +x install.sh && ./install.sh
+ - bii setup:cpp
+script:
+ - bii init
+ - bii open manu343726/oo-cmake
+ - rsync -av --exclude="blocks" --exclude="bii" --exclude=".git" --exclude=".gitignore" . blocks/manu343726/oo-cmake
+ - cmake -P build/script.cmake
+after_success: cmake -P build/after_success.cmake
+after_failure: cmake -P build/after_failure.cmake
+after_script: cmake -P build/after_script.cmake
+
+deploy:
+  provider: biicode
+  user: manu343726
+  password:
+    secure: ZBdHcI8HEKMSBZbx5xWT5d7iIzm6db8YDibt7TNC9xM3B30+/D+oFXkvMquxp2cYuhhU2cX+IAwLA4AhJmbYzDvUHnIA+7fJjO8DFngf933cv7lO+Wu/pbEN9gmn/nNxR9zZ1kQfH627gWaehq0MQEhXyLebTzDjY3u1h55PIpU=
+  skip_cleanup: true #The biicode block is generated during build, don't discard build changes!

 # branches:
 #   only:
 #     - master
-#     - devel
+#     - devel

There are changes on the README.md too, the travis badge points to my builds, there's a cool biicode badge I generated with shields.io, etc.

Since you have to perform some steps to accept this changes, not only to accept .travis.yml changes, I decided to put this as an issue with everything explained in depth instead of a pull request.

Let's see how you can setup your project to deploy a biicode block automatically:

How to create your oo-cmake block

You can download our client from here.

1. First create a biicode account.

2. Set up a new biicode project localy. This will be used to create and set up the block:

   $ bii init oo-cmake-setup
   $ cd oo-cmake-setup

   A folder oo-cmake-setup is created, with three directories inside:

   • bii/: biicode project metadata. Our .git/ counterpart.
   • blocks: Folder containing blocks being edited in the project. Each block has the path blocks/username/blockname.
   • deps/: Location of dependency blocks. Follows the same path convention as project blocks: deps/user/blockname.

   Say we have a block a created with the account user. That block has some sources depending on a block b created by developer. After running bii find (The command which searches and solves dependencies) our project looks like this:

    +-- oo-cmake-setup
    |   +-- bii
    |   +-- blocks
    |   |   +-- user
    |   |   |   +-- a
    |   |   |   |   +-- main.cpp
    |   +-- deps
    |   |   +-- user
    |   |   |   +-- b
    |   |   |   |   +-- foo.hpp

   In this example, main.cpp from a has an #include <developer/b/foo.hpp> directive.

   See getting started from our docs for more info about project layout.

3. Create the oo-cmake block:

   $ bii new toeb/oo-cmake

4. Create the biicode.conf file for the block, adding the explicit dependencies inside the [dependencies] entry:

   $ vim blocks/toeb/oo-cmake/biicode.conf

   ---

   [dependencies]
     oo-cmake.cmake + build/* cmake/* resources/* src/*

5. **Create a CMakeLists.txt for the block with the target disable:

   INIT_BIICODE_BLOCKS()
   ADD_BIICODE_TARGETS()
   set_target_properties(toeb_oo-cmake_src_cmakeJsonParser_cmakeJsonParser                                                      
                    PROPERTIES EXCLUDE_FROM_ALL 1 EXCLUDE_FROM_DEFAULT_BUILD 1) 

6. Publish the block:

   $ bii publish toeb/oo-cmake

   This will ask you for your account and password. Then the block is in the biicode cloud and we are ready to update it automatically via Travis CI builds.

   Updating the block with Travis CI

Take your .travis.yml and add the following entries:

install: 
- wget http://apt.biicode.com/install.sh && chmod +x install.sh && ./install.sh #
- bii setup:cpp
script:
- bii init
- bii open toeb/oo-cmake
- rsync -av --exclude="blocks" --exclude="bii" --exclude=".git" --exclude=".gitignore" . blocks/manu343726/oo-cmake

Now run travis setup biicode. It will add the deploy entry to the .travis.yml file:

deploy:
  provider: biicode
  user: toeb
  password:
    secure: YOUR ENCRYPTED BIICODE PASSWORD
  skip_cleanup: true #The biicode block is generated during build, don't discard build changes!

Note the skip_cleanup entry, it's very important! Since the block is opened and updated during build, we should specify Travis CI to not discard build changes before deploy. It should be added explicitly.

That's all!

Feel free to ask me whatever question you want to, use the comments system bellow if you like.

[toeb]

Hi!

First of all thanks for all the work that you invested. I'm happy that someone uses this library ;)

I'd be happy to add oo-cmake to bii. Currently I do not have the time to do so. I'll get back to you as soon as I can.

[toeb]

Hi I'm going to try doing what you described now. I think I want to deploy on changes to master and when I create tags. would I be correct in assuming that after deploy: ... I would add a on: \n tags: true 'n branch: master and this would update the biicode block correctly?

[toeb]

Also: I actually do not use the c++ project currently (maybe in future when I decide to generate cmake code would I start programming somehting in c++) to run oo-cmake all you need is cmake/* and resources/* am I correct in assuming so I would write the followng into the biicode.conf

[dependencies]
oo-cmake.cmake + cmake/* resources/*

[toeb]

so I created a pull request - would you mind checking it If its ok I'll merge into master

[Manu343726]

Hi!

I'm glad to see you tried biicode and integrated a block into your library!

About the projects using oo-cmake, I'm working in some internal projects involving complex cmake scripts, and I think your library may help us. I have no public examples using oo-cmake currently, but the rest of the biicode team is so expectant about your library. If oo-cmake integrates with biicode as expected (And our (mine and yours) blocks give the impression that will be true :) ) we will be using it as a main tool in all our cmake scripts.

[Manu343726]

Now your questions:

• Yes, that may configure the deploy as you like. Check Travis CI docs.

• What about the expression parser? Is not under the src/ directory? Please note that I'm not completely sure how your library processes your custom expressions. I'm correct? For example, I assumed that src/ was needed, that's why you should disable the generated target (The one generated from the JSON parser) build. In my block, I use a foreach() to disable all executable targets, since the oo-cmake block should be a CMake only block:

  foreach(executable ${BII_BLOCK_EXES})
    set_target_properties(manu343726_oo-cmake_${executable} PROPERTIES EXCLUDE_FROM_ALL 1 EXCLUDE_FROM_DEFAULT_BUILD 1)
  endforeach()

[Manu343726]

I'm currently working on adding Boost support for biicode, but I'm having some problems with large Boost compilation times. Calling Boost build from execute_process() produces no output at all during compilation, which is not fair from the UX point of view (Note that Boost build may take minutes, even hours).

Do you have any feature that may help with this, or at least makes possible to print a fancy progress text while running another command in background?

You can check the experimental Boost here. Note the scripts are under development (i.e. soooooooo ugly and full of sparse variables), but when the thing will work I have planned to simplify it via your library, with command wrappers, maps, etc. Please don't share the project, is still an experiment, too early to announce Boost support :)

Again, thanks a lot for all your feedback. All the biicode team will stay tuned to oo-cmake (if Boost with oo-cmake works, there are some of my mates talking about Qt via a oo-cmake based hook and more "big libraries", OpenCV, WxWidgets, etc). I hope oo-cmake will become a first-class tool for any cmake script. I really like it.

[toeb]

The expression parser is written in pure cmake (lang2, parse_script function using the expr.json as input file) and thus only needs the parser is also used to parse json (json.json)

so the only folder needed is resources.

I have written some helpers for processes see https://github.com/toeb/oo-cmake#parallel-processes the functions described can be used to start and wait for a process and for example emit a status indicator to the console.

sadly it is not completely (but almost) platform independent as it uses either bash or powerscript to start the processes. (untested on mac) (It could be really made platform independent using boost.process and a custom application)

maybe it'll help you ;)

[toeb]

travis does not deploy to biicode

[Manu343726]

Thanks! I was aware of your processes functionality, but I though the output was stored to a variable as execute_process() does. But if I'm allowed to do message() in a subprocess using your process_start_script() this will be perfect for me. Let's print a bee with oo-cmake while building Boost! cool.

I noticed your oo-cmake biicode block is broken (I'm trying to depend on it :)), since it has no sources inside. Are you maintaining block by hand instead of using travis?

As usually, thanks a lot for your feedback.

[toeb]

Yeah this issue should still be open - travis does not deploy to bii because Skipping deployment with the biicode provider because this branch is not permitted to deploy

I'm still trying to work that out.

process_start_script starts a cmake and hide the output - however the process is started asynchronously - so you can print anything as long as the process is running. also you can repeatedly read the stdout of the script you started. (Look at the unit tests to find out how this works)

[Manu343726]

You asked me for help to update the bii block on travis builds.

I think the best way is to use raw bii calls instead of the travis deploy service (It's a bit limited). Something like this:

language: cpp
os:
 - linux
 - osx
compiler:
  - gcc
matrix:
  allow_failures:
    - os: osx
env:
  global:
    # github api token and secret
    secure: "fWQnD0UqfX9SfctAu7lAnUJ4V5brp8UqdmPxQ67MlbmR5GAXQK9yezeQEsOnDCbihM6H/WTkonR920+B14fOb1NO0RSY39yHzjZG6B4tM6ZJwRW0CAlgFzlCqdwyL//7EhfKnxmWqflsA8oY+ihuZsj8P9ZtQSpNWENlu0Cl324="
    secure: "i9rwJYZa6VDCpjXFaJoUggdn12mC1WB2zb9zaFP57VocTyL37Kpt3btXb9De5VQEJbyVKG2b9zlts20dEqvvruXBBuhWi+15VICT1yqq3zDQt25EPPDXw+AcB58dG+0icY1VKklWXR2Qxms4x/EZSN4TDNUp6LZgh+ACWlE1SRU="
  matrix:
    - CMAKE_DIR="v3.1" CMAKE_VERSION="3.1.2" 
    - CMAKE_DIR="v3.0" CMAKE_VERSION="3.0.1" 
    - CMAKE_DIR="v2.8" CMAKE_VERSION="2.8.12.2"
    - CMAKE_DIR="v2.8" CMAKE_VERSION="2.8.7"
matrix:
  allow_failures:
    - env: CMAKE_DIR="v2.8" CMAKE_VERSION="2.8.7"
notifications:
  slack: fallto:MlnVOMNkx8YopsaSSxqh2Rcr
before_install:
  - sudo add-apt-repository --yes ppa:kalakris/cmake
  - sudo apt-get update -qq
  - sudo apt-get install mercurial
  - sudo apt-get install git  
  - sudo apt-get install subversion
  - git config --global user.email "travis-ci@example.com"
  - git config --global user.name "Build Server"
install:
  - wget http://apt.biicode.com/install.sh && chmod +x install.sh && ./install.sh
  - bii setup:cpp
  - bii init
  - bii open toeb/cmakepp
  - rsync -av --exclude="blocks" --exclude="bii" --exclude=".git" --exclude=".gitignore" . blocks/toeb/cmakepp > /dev/null
  #- sudo apt-get install cmake
  #- cmake --version
  # install cmake --
  - wget "http://www.cmake.org/files/${CMAKE_DIR}/cmake-${CMAKE_VERSION}.tar.gz"
  - tar xf "cmake-${CMAKE_VERSION}.tar.gz"
  - cmake -Hcmake-${CMAKE_VERSION} -B_builds -DCMAKE_BUILD_TYPE=Release -DCMAKE_USE_OPENSSL=ON -DCMAKE_INSTALL_PREFIX="`pwd`/_install"
  - cmake --build _builds --target install
  - export PATH="`pwd`/_install/bin:${PATH}"
  # -- end
  # verify installed cmake --
  - which cmake
  - cmake --version    
  #-- end

script: 
 - cmake -P build/script.cmake

after_success: 
 - "cmake -P build/after_success.cmake"
 - bii user toeb -p $YOUR_ENCRYPTED_BII_PASSWORD
 - bii publish toeb/cmakepp --tag STABLE
after_failure: "cmake -P build/after_failure.cmake"
after_script: "cmake -P build/after_script.cmake"

deploy:
  - provider: releases
    api_key:
      secure: Y8fO6kLjCkPHi7qa1tkv9kVKvoEe/VsRO9gY8LFFsBliKFD1DC/+Sf1l5ij3SGawFduzCWWk4O62lEVawEnUhOqHFb+D6+SExVzF5QvkgSvnp6BkpyosvCn0/FXBy4tt4KZyLSJ3CzqLvDnD99d7vouHyxiLZy4JSTjmlnklkAo=
    file: release/cmakepp.cmake
    skip_cleanup: true
    on:
      tags: true
      all_branches: true

Contributing on that script means I need your credentials to test publication. That's why I haven't sent you a pull request instead :(

Note how the block is published as STABLE. This allows anyone to use your block. By default blocks published as DEV cannot be reused by others except explicitly asked in user's policies.bii file.

You can also use travis branch variables to set up different biicode tracks for your block matching each development branch you have on the repo. Check the python scripts I have at boost-biicode to generate different block tracks from block templates. These scripts for easy block generation and publication will be released anytime soon as an independent project

[toeb]

so if I understand correclty: I need to create an encrypted environment variable called YOUR_ENCRYPTED_BIIPASSWORD which contains my password and then the script you posted will work .?

[toeb]

I must confess I haven't used biicode much (as I am focussing so much on cmakepp) so terminology like tracks escapes me currently. Are they the biicode equivalent of tags/branches?

[Manu343726]

The only difference between your current travis script and the one I posted above is the publication method. In yours the travis deployment service is used, what I propose is to use manual publish (Note the bii publish call on after_success) instead. Using bii directly allows you to specify the kind of publication: DEV (development, not intended to be used by users. It's the default mode). BETA, ALPHA, STABLE (It's a stable release, users are able to use the block without explicit configuration).

I think the best way to handle bii publication is to release as STABLE only on releases (That is, on git tagged commits). You can configure the travis script to run bii publish toeb/cmakepp --tag STABLE only on releases if you like. I didn't suggest that first since, if I got your dev workflow correctly, you only merge/push to master if the new features were ready and tested in depth.
Note bii publish command accepts an extra parameter to specify release version if you like. This is the help from bii publish:

usage: bii publish [-h] [--tag TAG] [--versiontag VERSIONTAG] [--msg MSG]
                   [--all] [-r [REMOTE]]
                   [block]

Publish (publicly) one or all the blocks of the current project.

positional arguments:
  block                 Block name, e.g.: bii publish my_user/my_block. Do not
                        use with --all argument

optional arguments:
  -h, --help            show this help message and exit
  --tag TAG             Release life-cycle TAG, e.g: bii publish --tag ALPHA
  --versiontag VERSIONTAG
                        Name tag for the version. e.g: v1.2 or "Sweet Beacon"
  --msg MSG             DEPRECATED: Publication description
  --all                 Publish all blocks. Do not use with block argument
  -r [REMOTE], --remote [REMOTE]
                        Publish VCS remote info. Format: "remote_url (branch)
                        @commit_id #tag" or blank to autodetect (currently
                        only git supported)

[Manu343726]

About tracks, you are correct, block tracks are our "branches" equivalent. The point is that you can maintain different tracks of a block, and later when you use the block you can specify what track you want via your dependencies settings (biicode.conf file, [requirements] entry).

As an example, we maintain an libuv block. There are three different tracks for that block representing the three different libuv versions we support. Another example is how I integrated cmakepp with boost-biicode hooks. What I did is to open your block, "fork it" i.e. create a new track from your block, and add the changes I need on my track of cmakepp. There's more information about tracks in the docs.

What I suggest is to match your dev workflow (becker + master branches) on biicode too, releasing in two correspondent tracks. This is only an idea, note is not necessary at all.

Don't worry about biicode usage, you are giving us a lot of useful feedback. Keep focused on your project, I know what maintaining a personal project on freetime means :P When cmakepp main features will be done, get a chance to test your lib with bii. You only have to include(toeb/cmakepp/cmakepp) in any cmake file of a block :)