Simplify building process by utilizing Conan dependency manager

[morevnaproject]

I think we can integrate with Conan C/C++ package manager to provide all required dependencies for building Synfig on any platform.

https://conan.io/

Benefits:

• Developers can easily set up build environment on any platform (and theoretically any build toolchain).
• This will also simplify packaging process (Appimages, Win installers, OSX App) and will allow to implement automatization for this process via CI.

Previously, I tried to achieve the same using NIX package manager, but at the end it turned out not a best fit for this task (in particular, it is hard to deal with switching versions for libraries; cases when you want to temporary switch library version for testing is hard to handle).

I am still in the process of investigating Conan, but I like what I see so far. This page is giving a good overview - https://docs.conan.io/en/latest/getting_started.html I am especially like the integration with BinTray ecosystem.

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/65531561-simplify-building-process-by-utilizing-conan-dependency-manager?utm_campaign=plugin&utm_content=tracker%2F321991&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F321991&utm_medium=issues&utm_source=github).

[rohits64]

Hi, I am Rohit from IIT Kharagpur. I would like to contribute to this project in Gsoc. Currently, I am going through the given doc( https://docs.conan.io/en/latest/getting_started.html). What are things to be done before submitting the proposal?

[morevnaproject]

Hi @rohitsanjay! My name is Konstantin Dmitriev, I will be mentoring this project. Can you please give me some information about your skills and background? Also, please let me know which operation systems you are capable to work with (Linux, Windows, OSX)?

[morevnaproject]

About the work on this issue:

I suggest to make some preparation work, to proof your capabilities to work on this project.

Synfig depends on various libraries, that required for building (see https://synfig-docs-dev.readthedocs.io/en/latest/common/dependencies.html). Some of those libraries are already available as Conan packages, some are not. If some library is not available as Conan package, then we need to pack it.

So, as first step, we need to create a list of all required libraries. In this list we will mark which libraries are Conan-packed and which are not (this is going to be our TODO list and it will show the expected amount of work).

Let's start with a list of dependencies for synfig-core: please examine https://github.com/synfig/synfig/blob/master/synfig-core/src/CMakeLists.txt and make a list of required libraries.

Same work should be done for synfig-studio.

[rohits64]

@morevnaproject I am second-year computer science undergraduate at IIT Kharagpur, India. I have prior experience with C/C++ and python. I use both Ubuntu and Windows as per requirement.

[rohits64]

I have made a list and verified if they are Conan packed or not. The list for synfig-core is followed:

• [x] cmake(>=3.1)

• [x] boost

• [x] zlib

• [x] threads(system)

• [x] sigc++

• [ ] glibmm

• [ ] giomm

• [x] cairo

• [x] libxml

• [ ] mlt++

• [x] fftw

• [x] pango

• [x] imagemagick

• [x] git(system)

  from https://github.com/synfig/synfig/blob/master/synfig-core/src/modules/CMakeLists.txt

• [x] freetype

• [ ] std

• [ ] bmp

• [ ] dv

• [ ] example

• [ ] ffmpeg

• [ ] filter

• [ ] geometry

• [x] gif

• [ ] gradient

• [x] jpeg

• [ ] noise

• [ ] particle

• [ ] png

• [ ] ppm

• [ ] svg

• [ ] yuv420p mod_libavcodec - build failure mod_magickpp - made optional mod_mng - missing include mod_openexr - missing include mptr_mplayer - "This code has vulnerabilities"

[rohits64]

@morevnaproject The list for synfig-studio is:

• [x] cmake
• [x] gettext(system)
• [x] sigc++
• [ ] gtkmm
• [x] libxml

Here in https://github.com/synfig/synfig/blob/master/synfig-studio/src/CMakeLists.txt contain packages which are common with synfig-core, which are sigc++ and libxml. I don't think there is a need to double check the same module, we should remove those line from synfig-studio code.

[morevnaproject]

@rohitsanjay Looks good!

Just some quick notes about synfig-core:

• "threads" is a system lib, so there is no need to package it.
• There are also addition dependencies for modules listed here - https://github.com/synfig/synfig/blob/master/synfig-core/src/modules/CMakeLists.txt

For synfig-studio: I am strongly suspect that for "gettext" we should also use a system one (otherwise, this looks weird why Conan team has not packed it yet).

Here in https://github.com/synfig/synfig/blob/master/synfig-studio/src/CMakeLists.txt contain packages which are common with synfig-core, which are sigc++ and libxml. I don't think there is a need to double check the same module, we should remove those line from synfig-studio code.

Agree. Let's try to remove that. ^__^

[morevnaproject]

• "cairo" available as Conan package here - https://bintray.com/bincrafters/public-conan/cairo%3Abincrafters
• "sigc++" is also has Conan package - https://bintray.com/bincrafters/public-conan/sigc%2B%2B%3Abincrafters.

Also, it's ok to take "git" package from system. ^__^

[morevnaproject]

• "glibmm" packed for Conan - https://gitlab.com/no-face/glibmm-conan

[rohits64]

@morevnaproject Thanks for the help I will recheck and edit the required dependency and report to you.

[morevnaproject]

@rohitsanjay As a next step I suggest to create a Conan package for "pango". I also recommend to mention that you work on it here - https://github.com/conan-io/wishlist/issues/17

[rohits64]

@morevnaproject I have updated the list, Please verify, I have little doubt about it. Thanks,

[morevnaproject]

@rohitsanjay Just want to note that Synfig's CMake scripts are not mature yet (we are in transition from Atuotools to CMake), So, they seems to be not listing all dependencies. So, I have consulted https://github.com/synfig/synfig/blob/master/synfig-core/configure.ac to complete the list.

Here are my comments:

• "std" - this is okay to remove (no dependencies on external libraries).
• "bmp" - seems to be no dependencies on external libraries as well.
• "dv" - this is okay to remove (no dependencies on external libraries).
• "example" - no dependencies on external libraries
• "ffmpeg" - needs "ffmpeg", available as Conan package
• "filter" - this is okay to remove (no dependencies on external libraries).
• "geometry" - this is okay to remove (no dependencies on external libraries).
• "gif" - this is okay to remove (no dependencies on external libraries).
• "gradient" - this is okay to remove (no dependencies on external libraries).
• "noise" - this is okay to remove (no dependencies on external libraries).
• "particle" - this is okay to remove (no dependencies on external libraries).
• "png" - libpng is available as Conan package.
• "ppm" - this is okay to remove (no dependencies on external libraries).
• "svg" - this is okay to remove (no dependencies on external libraries).
• "yuv420p" - this is okay to remove (no dependencies on external libraries).
• "mod_libavcodec" - should build fine with FFmpeg > 3
• "mod_magickpp" - depends on "imagemagick" (?), available as Conan package.
• "mod_mng" - depends on "libmng" - http://www.libpng.org/pub/mng/
• "mod_openexr" - depends on OpenEXR, available as Conan package - https://github.com/conan-community/conan-openexr
• "mptr_mplayer" - It's okay to forget about it. ^__^

[morevnaproject]

P.S. Just to avoid misunderstanding: When I say "this is okay to remove" I mean "remove from list". ^__^

[rohits64]

Thanks, @morevnaproject As a summary we need to create Conan package for: giomm, mlt++, pango, gtkmm as the priority. I will be working on pango now.

I have made a list and verified if they are Conan packed or not.

• [x] cmake(>=3.1)

• [x] boost

• [x] zlib

• [x] threads(system)

• [x] sigc++

• [x] glibmm

• [ ] giomm

• [x] cairo

• [x] libxml

• [ ] mlt++

• [x] fftw

• [ ] pango

• [x] imagemagick

• [x] git(system)

  from https://github.com/synfig/synfig/blob/master/synfig-core/src/modules/CMakeLists.txt

• [x] freetype

• [x] ffmpeg

• [x] jpeg

• [x] png

• [x] libavcodec (included in ffmpeg 4.1)

• [x] mng

• [x] openexr

The list for synfig-studio is:

• [x] cmake
• [x] gettext(system)
• [x] sigc++
• [ ] gtkmm

Not required:

• [ ] std
• [ ] dv
• [ ] filter
• [ ] geometry
• [x] gif
• [ ] noise
• [ ] gradient
• [ ] particle
• [ ] ppm
• [ ] svg
• [ ] yuv420p
• [ ] mplayer

[morevnaproject]

@rohitsanjay You can remove 'bmp" and "example" as well (no need for Conan packages). ^__^

Also, I suggest to remove everything which is falls into "Not required" category.

In all other aspects I am happy with this list. ^__^

[morevnaproject]

Hi @rohitsanjay! Any progress on this? ^__^

[rohits64]

Hi @morevnaproject! , Sorry for the delay from my side, I was ill from the past 10 days so I was unable to do work. I am almost okay now I expect to start working at my normal pace from early this weekend. Thanks!

[morevnaproject]

Great, I am happy to hear you are better now. ^__^

[rohits64]

Thanks, @morevnaproject I am now on this project. The workflow of this project should is to create packages for four libraries, giomm, mlt++, pango and gtkmm. And then make recipe for synfig with help of Conan. I was going through documentation of Conan and watching video related to packaging libraries, I was having some difficulty if you have some resource it would be very helpful.

[morevnaproject]

OK, I am looking forward for pango package. ^__^

[morevnaproject]

I was going through documentation of Conan and watching video related to packaging libraries, I was having some difficulty if you have some resource it would be very helpful.

This page is your best friend - https://docs.conan.io/en/latest/creating_packages.html

Also, you can see how other similar libraries are packages. For example, cairo - https://github.com/bincrafters/conan-cairo

[rohits64]

Also, you can see how other similar libraries are packages. For example, cairo - https://github.com/bincrafters/conan-cairo

Thanks, @morevnaproject I was looking for this kind of thing.

[Croydon]

@rohitsanjay Also have a look at our Bincrafters templates 😄

https://github.com/bincrafters/conan-templates/

We are maintaining ~ 450 packages so they are pretty battle tested

[rohits64]

Thanks, @Croydon

[rohits64]

@morevnaproject
Hi, I have submitted the proposal draft of Gsoc, please review it. Thanks

[morevnaproject]

Hi! Sorry, got your message a bit late and wasn't able to review yesterday. Your final proposal is passed to our tea for review now.

[rohits64]

No problem @morevnaproject, looking forward to working you.

[morevnaproject]

BTW, are there any progress on Cairo package? ^__^

[rohits64]

I was working on Pango package, I have created a reprository on Pango and working on it. Due to University exam, I am getting very less time. Once it ends I expect faster development on it

[morevnaproject]

Ah, yes, sorry, I meant Pango.

Please consider that this "proof" task should be finished by April 20th - after that we will be taking final decision on which projects to accept. ^__^

[rohits64]

Okay, I will try my best.

[ivan-kulikov-dev]

https://github.com/conan-io/conan-center-index

Conan has switched to a new package adding system. When a new package is added, it runs on ci

Try try add openexr 2.4.0 in conan-center-index now. https://github.com/conan-io/conan-center-index/pull/204

[Croydon]

Before working on a new recipe please have a look first into:

https://bintray.com/conan/conan-center Update: https://conan.io/center https://github.com/conan-community/ https://github.com/bincrafters

to avoid working on something which already exists.

All recipes which are either in conan-community or Bincrafters but not the Conan Center Index will be transferred eventually.

Thanks for contributing to Conan!

[morevnaproject]

Hi!

Information for aspiring students/contributors, who would like to take this task as part of GSoC:

You should start digging this task BEFORE gsoc starts and show some results BEFORE May 10 2022. Your results will be reason to decide if you're going to be accepted or not.

I would like to clarify next steps to proceed with this issue, so you can start digging already.

1. I have draft PR with basic conan configuration file (conanfile.py) added - https://github.com/synfig/synfig/pull/2286
2. Next goal is to make synfig-core compile using conan packages.

For that we have to examine dependencies of synfig-core and add all required conan packages to conanfile.py (see https://conan.io/center/ for list of available conan packages and their versions).

At the same time, we will find which dependencies are not available as conan packages yet. The next step will be to pack those dependencies as conan packages.

Here's how to find minimal required dependencies of synfig-core:

1. Create a virtual machine with a clean Linux system and minimal develpoment tools (gcc compiler, git, conan, etc. but not install headers for dependent libraries!)
2. Clone synfig's repository
3. Execute conan install from synfig's code directory to install dependencies listed in conanfile.py.
4. Try building synfig-core (use command ./2-build-production etl && ./2-build-production core)
5. You'll get error messages when some package is missing. Add it to conanfile.py.
6. Repeat from step 3.

[rock16]

Hi! @morevnaproject My name is Rock. I am actually interested in working on this project ideas as my gsoc 2022 project since I have the required skills for this project. I have actually made a few contributions to synfig also.

I would like to clarify next steps to proceed with this issue, so you can start digging already.

I have draft PR with basic conan configuration file (conanfile.py) added - https://github.com/synfig/synfig/pull/2286 Next goal is to make synfig-core compile using conan packages.

like you mentioned, I will like to start working on this task. Though I noticed that your PR has not been merged to the master branch so I intend to create a new branch and adding the conanfile.py file to the branch to start working on this task right away.

[morevnaproject]

Hi @rock16! Yes, the PR #2286 is not merged, but I recommend you to use its branch as a starting point. You can checkout my branch (https://github.com/morevnaproject/synfig/tree/conan) and start from there.

[rock16]

Hi @morevnaproject I would like to know if we are going to have a CI system because that is were I believe the real benefits comes from since for every package I create for instance we should be able to have it on a repository such as conan. I would also like to include it in my Gsoc proposal too.

[morevnaproject]

Yes, when you create a list of required packages, you will find that some packages are missing in Conan. And you will need to pack those dependencies for Conan. So, do you mean to setup CI for each new Conan package?

[rock16]

Hi @morevnaproject

Yes, when you create a list of required packages, you will find that some packages are missing in Conan. And you will need to pack those dependencies for Conan.

If I understand properly, lets say I create a conan package for glibmm on my local machine, I will have to upload the package and the recipe to conan repository right?

My thoughts were that we would use a CI server like jenkins to build our libraries and generate the conan package so that each time we make changes to the recipe for example. we just have to git commit and the libraries get re-compiled automatically and pushed to conan repository. So I wanted to include this in my gsoc proposal.

I actually understand that for the moment I need to create those missing packages and its dependencies that are missing in Conan on my local virtual machine.

I have also submited a draft of my proposal. Is the project size Medium or Large?

[morevnaproject]

Yes, we will need a CI for generating packages and also testing how they build in different environments - Linux/Windows/MacOS. We also have a custom Docker image with an environment for cross-compilation of Synfig (and its dependencies) to various platforms - https://github.com/morevnaproject/morevna-builds. We use this image to generate installers for Windows and appimages for Linux. It would be nice to test new Conan packages in this environment too.

I suggest to use Github Actions for CI.

Regarding the project size, that depends on the amount of work you plan in your proposal and the amount of time you can dedicate to the project during this summer.

[rock16]

Thank you very much for your support. Right now I'm working on the recipes for dependencies missing on conan center.

[theartful]

Hi, I'm glad to be accepted to work on this project :)

This is the current progress porting dependencies to conan:

1. libsigc++-2
2. glibmm-2.4
3. cairomm-1.0
4. libxml++-2.6

Next steps:

1. Porting pangomm
2. Porting gtkmm 3
3. Porting mlt++
4. Enabling ImageMagick on Windows

Synfig-core still has dependencies for mlt++ and ImageMagick. Conan has a recipe for ImageMagick but it does not work on windows. However, building synfig-core using conan should be doable right now.

But I think first, I will submit a PR that improves the cmake build scripts to customize building the different components of synfig alone (ETL, synfig-core synfig-studio).

[ice0]

@hesham-essam

As far as I know, glibmm-2.68 is not compatible with GTK3. It is for GTK4. You must use glibmm-2.66 for GTK3.

[theartful]

@ice0 Yeah, sorry I miss-wrote. I added support for both glibmm-2.68 (with version 2.72.1) for gtkmm4 and glibmm-2.4 (with version 2.66.4) for gtkmm3.

[theartful]

I made a PR for pangomm: https://github.com/conan-io/conan-center-index/pull/10992

[theartful]

gtk3 does not build on windows using conan, so I will fix that, then I will start working on porting gtkmm 3

[theartful]

gtk 3 on windows: https://github.com/conan-io/conan-center-index/pull/11013

[theartful]

Initial draft of building synfig with conan: https://github.com/synfig/synfig/pull/2667

[theartful]

I'm currently working on an ImageMagick recipe (as the one in conan currently does not work on Windows), and I will make a PR for it in the conan repo once finished, hopefully today or by tomorrow, then I will focus on making the conan PR reach the same state as the vcpkg one.