Migrate the dependency management to Conan

[cqjjjzr]

Hi.

I'm the author of the time/align function and I found you merged my function. Thanks a lot!

When I wrote the function, I found the dependency management of Aegisub so painful(especially for Windows, since they used Visual Studio to build all dependencies and those .vcxprojs have problems for Visual Studio 2019), so I'm now working on migrating all the dependencies to Conan to avoid manual dependency configuring.

After a little bit of research I found some libraries available on conan-center and bincrafters/public-conan, while others are not ported for Conan so we must write our own Conan recipes for them.

Are you interested in cooperating on this?

Cheers,
Charlie Jiang

[wangqr]

Thanks for the time/align work. We use it to fix esrXP misdetections, and it's really handy!

Yeah the dependencies are hard to manage, so I've ported the whole project to CMake. So now this project only builds Aegisub. All dependencies are automatically located by CMake, therefore updating dependencies are quite easy. And on linux, libraries installed by disto package manager can be automatically used. Building on windows and linux have been tested and have no problem currently.

Installing dependencies on Windows might be a little difficult, and some may need build from source, namely libboost whose official binary build has no ICU support, libass, and libiconv. But this only needs to be done once.

Given I have no problem building it with latest dependencies, I'm only maintaining this in my spare time, and I haven't used Conan before, I may not have time to try out Conan. But feel free to contact me if you encountered any difficulties in building Aegisub. Good luck!

[wangqr]

此外，我会中文

[cqjjjzr]

好的。

目前的确问题最大的方面就是在Windows上进行构建。Aegisub 官方仓库提供的Windows构造需要通过 git submodule 拉取所有依赖的源代码，再通过 由Aegisub维护者提供（而非这些依赖库的维护者提供）的Visual Studio构造配置（这使得迁移依赖版本极为困难）来进行编译。

我们将构造工具转移到Conan的工作将会在近期完成，届时可以再进行联系。

此外，非常感谢你们在将Aegisub本体的构造迁移到CMake方面的工作，这极大地减小了我们的工作量。

Cheers,
Charlie Jiang

[cqjjjzr]

另外，我发现Aegisub的官方仓库的依赖构建中有一些非默认的构建开关，这在构建 wxWidgets 与 FFmpeg上极为明显。但是我在分析哪些是必要的库时遇到了困难。特别是wxWidgets，在其更新后其中某些组件的构建似乎有些难以理解，例如 expat 与 scintilla。

在这几个方面你有什么相关的可供参考的信息吗？

Thanks,
Charlie Jiang

[wangqr]

印象里有的子项目是故意抠出了需要的文件来构建的，以减小最后静态链接的exe大小。所以实际上直接使用默认编译或者官方二进制也是可以的，无非就是产物大一点（如果动态链接并且只打包所需的dll的话甚至大小差别不明显）。不太确定expat是啥。scientilla是wxStyledTextCtrl，它现在应该是core的一部分。

ffmpeg不是aegisub的直接依赖，它是ffms2的依赖，aegisub只依赖ffms2。

[cqjjjzr]

目前Aegisub所有依赖的Conan迁移接近完成。剩余没有完成的包有 luabins 和 csri，其中我仍准备让csri包保持嵌入项目的形式，而非创建一个Conan Recipe。这样做是否恰当?

另外关于 Lua，我目前不是很清楚Aegisub如何提供lua运行时。我只在vendors中找到了luajits并在你的CMakeLists.txt中找到了构建代码。我根据luajits的官方Makefile与另一位开发者（int010h）的脚本制作了LuaJIT 2.0.5 的Conan Recipe：（https://github.com/cqjjjzr/conan-luajit）。然而我在另一个Issue #15 中注意到你使用了额外的构建参数来提供兼容性。这是否会影响到我的Conan Recipe的构建？以及，Aegisub现行代码中是否包含对lua原版运行时的依赖（https://github.com/lua/lua）？以及moonscript等库是通过何种方法引入Aegisub的呢？其会影响到Conan Recipe吗？

[wangqr]

内嵌csri我觉得没问题

aegisub就是内嵌了lua解释器（设置了若干函数然后去跑用户脚本）。理论上可以使用任何版本的lua，只要脚本兼容（参见 63f4bf1beb824ab95c1fa281253c437cd8af5c4d ），目前由于有用到luajit ffi，所以迁回lua需要一定工作。 cmake里面编译luajit是抄的原Makefile，唯一修改是设置额外的LUAJIT_ENABLE_LUA52COMPAT。首先lua是一个没有小版本间兼容性的语言（lua51和lua52互不兼容，lua52和lua53互不兼容，然后lua的各版本用户量也很碎片化）luajit标称兼容lua51，打开此开关可牺牲lua51兼容性换取对部分lua52特性的支持（ http://luajit.org/extensions.html ）。我目前还不确定原作者有没有在脚本中使用这些特性。如果没有，那么使用未开开关的luajit说不定也可以，代价可能是极小部分的用户贡献插件用不了了。 luajit是lua的一种实现，aegisub不依赖原版lua。luabins等c写的库是设置lua环境时加载的，moonscript等lua写的库是放在 automation/ 里面由解释器去加载的

[cqjjjzr]

LUAJIT_ENABLE_LUA52COMPAT这一开关涉及到LuaJIT本体的编译吗？（否则我可能需要调整LuaJIT的Conan Recipe来允许增加这一开关）

另外，目前Aegisub提供的FFmpeg的构建configure参数采用了非常省空间的一些配置，我是否应该调整FFmpeg Conan Recipe来引入这些参数？因为如果相较于Aegisub最后一个release产生了巨大的体积膨胀似乎并不是一个好现象

[wangqr]

涉及，这一开关就是编译LuaJIT本体时使用的

ffmpeg不是aegisub的直接依赖，它是ffms2的依赖，aegisub只依赖ffms2

ffms2只用来加载视频，所以输出/编码之类的功能可以都舍弃掉，这是之前Aegisub的配置干的事情。由于目前我直接使用的ffms2预编译包，所以这些配置不再由Aegisub控制。

个人倾向于直接使用上游。目前我编译的所有windows版除csri和luajit仍用自带的外，其他依赖均使用原版上游，总大小才85M，对于目前的使用者应该关系不大。另外在linux上使用系统依赖相比于自己静态链接依赖更省空间，因为可以与其它程序共用（不太确定Conan是如何处理依赖的）

附一张最近的编译产物的磁盘占用：

[cqjjjzr]

好的。目前一个严重问题在于FFmpeg采用任何非上游的版本都会在Conan处触发重新编译，而在Windows上编译需要额外的msys2或cygwin等环境。个人觉得这样就失去了Conan管理依赖的意义（一键下载依赖），但是上游的FFmpeg包含了一大堆libx264等完全没有必要的外部库。我可能需要自己弄一个ffmpeg-minimal包...

Conan会使用静态链接直接把依赖链接进去，因此没有系统管理依赖的优势。但是好处是Windows和Linux有了统一且一键解决的依赖配置

[cqjjjzr]

已经成功进行了移植，可以编译并成功运行。在Windows下Aegisub.exe与ffms2.dll（除此以外我全部使用了静态链接）的体积总和为 57.2MB。

另外，我将原有的基于libresrc的引入资源文件的方式改成了直接通过Windows上的RC资源编译器加载，避免了在编译时使用lua。

一个问题：不知道现有版本有无此问题，在 888be0607fdfc2ed7801fd67f46ab4f98257d419 中你做了如下修改：

    toolBar->ClearTools();
+       toolBar->AddSeparator();
    toolBar->Realize();

这导致了在我更换构建系统之后的Windows版本上出现堆栈溢出。

其似乎是因为布局引发事件，事件引发布局变化导致的无限递归。

将会在晚些时候给出复现方法。

您可在 https://github.com/cqjjjzr/Aegisub/tree/dev-cj 查看我的修改。

[cqjjjzr]

I will complete the port for Linux and macOS soon, and if it could be a choice I'd like to merge it into the current dev branch because it provides a handy "one-click dependency management".

[wangqr]

I believe that on *nix system, the conventional way is to use system package manager to manage dependencies. So I'm not sure if it is proper to include a dependency management in a project. Just like no one includes anaconda stuff in python libs. Any packaging script for certain linux distro is also not included here, as they should be maintained by distro itself.

[cqjjjzr]

In the past, yes. But IMO, it's really not a good choice to ask users to do a lot of

apt install xxxx xxx xxxx

staff. Also, some packages' names may vary between different distros, causing inconvenience. However, althought conan is also a package manager, the cross-platform compatibility is better and is better associated to the build system.

As for anaconda, conan is totally different for it. Conan is a much more lightweight tool.

However, since it's a convention staff(ahh tradition), it's up to you to decide the way to handle deps.

[wangqr]

it's really not a good choice to ask users to do a lot of

Aegisub is already in most distros. The only thing a user needs to do is apt install aegisub. For developers who wants to build from source, resolving dependencies is a must-have skill, and they can decide whether use any package manager to fetch them, or build it from source. They can also use Conan, of course.

[cqjjjzr]

Of course this Conan-port is dev-oriented. So I think we can make a choice in the CMake script, asking if the user want to configure the dependency manually or use conan(However the Conan should be the only choice on Windows, and macOS probably). Is this OK?

[wangqr]

Actually CMake is already somewhat heavy, and I'm not sure if it will be adopted by upstream.

Quote from https://github.com/wangqr/Aegisub/issues/13#issuecomment-533939746 :

If the upstream is actively maintained again, I'll just leave this fork here and make PRs to the upstream.

And the CMake stuff will just be deprecated.

[wangqr]

As for Windows, there are also more widely used dependency managers IMO. E.g. MSYS2/MinGW or Microsoft/vcpkg.

[cqjjjzr]

Well, you mean all we shoule use those autotools staff?

I've already reviewed MSYS2 and vcpkg and a lot of dependencies are missing. So it's not a option.

Also, as you know, downloading some deps(namely ffmpeg) is extremely painful in China for some reason, and many members of my team meet problems about resolving dependency and upgrading the vcxprojs on Windows. It's absolutely wrong to include all those deps to one single .sln because this will load a lot of unnecessary stuff and slow down the whole Visual Studio.

So I think at least on Windows, CMake is necessary.

[wangqr]

The decision to download dependencies using submodule and use self-maintained vcxprojs to build them is already really ugly IMO. The cmake/find*.cmake stuff is also ugly, but at least they have nothing to do with how to build the dependency. The conventional way on any system is the developers should get those dependencies by whatever way they want, and the only thing the build system shuold care about is where is the include path, and where is the lib to link to. Anything related to how to build dependencies should not be here.

If a dependency is missing in a PM, they should be added separately, not in Aegisub's repo.

[wangqr]

You may want to open a issue at Aegisub/Aegisub, as this repo will not be maintained if a new windows build comes out from upstream. I am already working on rebasing on top of the current upstream, and the current dev branch may be renamed and left here. Or feel free to maintain your own fork.

[wangqr]

Let me make this clear.

In a project, things related to the build system should be included. This includes finding and linking to correct dependencies, and building Aegisub itself.

Things related to package management should not be included. This includes downloading and building dependencies. These should be maintained by corresponding package maintainers. And if certain build options is needed, it should be maintained by the package maintainer for that certain platform, like in Archlinux has PKGBUILDs, and debian has its own fork.

Under this criteria, anything in vendor including lua, and cajun should be dropped, as well as vcxprojs for building all these dependencies. We may want to make a exception here for csri as it is not maintained as separate project, it won't update anymore, and it only defines a interface. The provided Visual Studio project does not meet this criteria. And the autotools stuff is much better than VS sln in this sense. The CMake is mostly written according to autotools.

[cqjjjzr]

So can I make these conclusions?

• We need to create a separate repository named "Aegisub-Deps", and put all those conan stuff(or apt yum pacman?) there.
• We need to completely switch to CMake and improve the CMakeLists.txt according to the existing autotools scripts.
• We need to supply all those modes in the CMake of the main repo:
  1. Manually configure: the developer should specify all dependencies from CMake options(find_package has already supported this)
  2. Find from local directories: exactly what find_package does currently.
  3. Load from CMake: load all configs from external CMake file. This provides support for Conan, and any other dep package manager supporting generate a CMake file.
• Maybe a submodule from main repo to the deps repo?

[wangqr]

If I read the documents correctly, here is the sample of creating conan recipes. Just like any other distro packaging stuff, each package should contain only the build script and essential patches.

[cqjjjzr]

Yes. I've created/found all the conan package recipes required by Aegisub. You can find them in my GitHub repository and in https://github.com/cqjjjzr/Aegisub/blob/dev-cj/dependencies/conanfile.py

[cqjjjzr]

Just like any other distro packaging stuff

You can also upload the prebuilt binaries. However since I couldn't create my AppVeyor for unknown reasons, prebuild pkgs for Windows are absent.

[wangqr]

and looks like this is the guide to create recipe. The recipe is a small folder which does not contain the original repo.

[cqjjjzr]

and looks like this is the guide to create recipe

Excatly. Also, conan itself don't contain any build system. Instead, it supports various build systems through command-line or some util classes like AutotoolsBuildEnvironment or CMake.

And the original code is downloaded in the source() method of a ConanFile object.

[wangqr]

So ideally you should create a similar repo somewhere (like on Ubuntu we have ppa source, on Arch we have AUR). The repo has a recipe folder, which contains one sub-folder for each dependencies of Aegisub, and one sub-folder for Aegisub itself. No code is contained in the repo, except conan recipes.

[cqjjjzr]

No. The conan recipes should be managed through Bintray.com and all those recipes and prebuild packages are uploaded there, and downloaded according to conanfile.py

[wangqr]

Well, if bintray is something similar to AUR, then you'll just create a recipe for Aegisub, and upload it there.

[cqjjjzr]

Excatly. I've already done. However, should we create a Conan recipe for aegisub? I believe Aegisub itself won't be depended on by any other packages, so it's unnecessary to create a Aegisub recipe. Instead, I've created recipes(and some recipes are already created by others) for packages that aegisub depends on.

[wangqr]

I believe that's how package managers work. They manage packages which can depend on other packages. They are not meant to only manage libraries or dependencies. At least this is how package managers work on all linux distros / mac (ports, homebrew) / windows (chocolaty).

I haven't checked the policy for bintray yet.

[cqjjjzr]

As far as I know, no one has uploaded application packages that are not related to building (e.g. cygwin 7z msys2) other packages to the conan-center or other major repositories. This also stands for vcpkg, but is different from chocolatey.

[wangqr]

vcpkg (and I would say NuGet) are designed for Windows. And the Windows-way to distribute an application is packing all dlls / whatever dependencies in a huge setup exe or msi. Although vcpkg claims to be cross-platform, it actually makes little sense using it on non-Windows systems, unless you want to create a virtual env like anaconda.

These two are designed to manage dependencies, therefore they can be used with minimal extra effort. Take vcpkg for example, if a dependencies is installed, CMake will just find them automatically, so the only thing needs to be done when building is to select dependencies, and no modification is needed on the build script of the application. No vcpkg specific script is ever needed.

Not sure if conan goes the same way.

[wangqr]

But if conan goes the same way as vcpkg, then the same thing also applies to conan:

• It should not be used on non-Windows systems
• It should not need any tool-specific script

[wangqr]

A quick search shows that there are some application packages on bintray 1 2. Not sure if this is conventional.

[cqjjjzr]

1. Although I use vcpkg little, I think vcpkg is redistributing build scripts instead of a huge msi.

2. vcpkg don't support a self-maintaining repo like Bintray, so if we have some packages that are not included in the vcpkg repo, we need to create PRs. However for conan, just upload them, and you can even distribute binaries.

3. conan is fully cross-platform, and I've tested all recipes created by me on Windows, Linux and macOS. However I don't know if you consider it making sense on *nix systems.

4. Unfortunately conan requires a little bit of tool-specific script. It has several generators for CMake. The cmake, cmake_multi, cmake_path and cmake_find_package generator. All of those need extra effort.

   • The cmake generator requires you to include() the generated .cmake file and run conan_basic_setup(). The include dirs will be automatically set, and you need to use ${CONAN_LIBS} in target_link_libraries().
   • The cmake_multi generator is similar to cmake generator, but generates for multiple build types (Debug, Release).
   • The cmake_path generator requires you to set it as CMake toolchain file or include it, and then you're able to use the legacy find_package() with your FindXXX.cmake files.
   • The cmake_find_package generator will generate several FindXXX.cmake, in replace of your existing FindXXX.cmake

   The latter two generators may be better meeting our needs. You can write a .ps1 and a .sh with them.

5. You will notice all those packages are not in the conan-center repository, and major community repos like conan-community and bincrafters have few application packages, so I believe it's not conventional.

[wangqr]

The cmake_path generator requires you to set it as CMake toolchain file or include it, and then you're able to use the legacy find_package() with your FindXXX.cmake files.

Yes this is the conventional way. User will install all dependencies in conan, set -DCMAKE_TOOLCHAIN_FILE=<conan_toolchain_file>.cmake when calling CMake, and everything should just work.

[cqjjjzr]

OK so after I complete the Conan repo what should we do on the current dev branch?

Also, you should avoid building luajit anymore. I believe we should switch resource management to use RC on Windows, and let the files be external instead of embedded to the elf on Linux, and use bundle on macOS.

[wangqr]

I don't plan to do major changes in dev branch by now, bug fixes still welcome. I am still waiting for the upstream binary release. If it comes out, this repo will be deprecated (not maintained). I might or might not rebase the CMake stuff on top of current upstream master, but right now I don't see much benefit. If I do the rebase, then improvements will be accepted in that branch, instead of dev.

I believe we should switch resource management to use RC on Windows, and let the files be external instead of embedded to the elf on Linux, and use bundle on macOS.

This is an improvement, not an issue. So it will have low priority, or I may never do this. Plus, I have no plan making linux or macOS builds, because for linux distros they already include aegisub, and I don't have a macOS computer to do the build.

[cqjjjzr]

For the CMake stuff, it may be better to completely deprecate the vcxprojs and autotools stuff and switch to a cross-platform build system. It may be CMake, or mason or anything other. Reasons are not only the Windows(msys2 is wayyyyyyyy heavier than CMake), but also the IDE.

For the Linux and macOS stuff I can help.

The RC res management is already done.

[wangqr]

For the CMake stuff, it may be better to completely deprecate the vcxprojs and autotools stuff and switch to a cross-platform build system.

The current build system ain't break, especially the autotools works with no issue IMO. Actually the autotools is still in travis CI of this fork. As long as there are people who can maintain it (tgoyne hopefully), that is fine. I made the CMake stuff, because I cannot maintain the VS project, and I need Windows build.

For the Linux and macOS stuff I can help.

It is OK to just do the release on your fork, or the upstream. I mean, my fork isn't special. I just maintained this for about half year, and the upstream has come back.

[cqjjjzr]

The current build system ain't break,

Actually I'm not fix it. Switching to CMake/Mason/Gradle/Ninja is kind of improvement.

[wangqr]

... is kind of improvement.

At least in my opinion, it is just a personal favor, or toolchain war. Non of the VS sln or autotools is dead. They are still widely used, with no major drawbacks. There might be some problem on how we use VS in this specific project, but that doesn't mean switching to another build system will improve the situation. At least for now, the CMake stuff is lacking multiple functions even on Windows, including generating translation and building the installer.

[cqjjjzr]

All of these fixes are already on my TODO-List. I'm now working mainly on completing and improving CMake files.

[cqjjjzr]

Things are getting worse when it comes to those complicated scripts like generating pot files, for CMake. CMake has too little ability to handle complex string processing. I'm now extremely regretful for not switching to Meson. What's your opinion?

[cqjjjzr]

Also I've already done the switchable dep management using CMake. You can check it out at https://github.com/cqjjjzr/Aegisub/tree/dev-cj . The README is updated.

The main idea is to provide a default dep resolving script using find package, and change the script to the one in another repo when we want to use conan. I believe this realized "decoupling dep management from build system"

[cqjjjzr]

Besides I noticed that building the translations and installers aren't functions provided by the VS sln. They are separated sh files. All functions provided by VS toolchain is now implemented in CMake.

[cqjjjzr]

I've finished the translation build scripts and switched your unit test scripts to use CTest(via GoogleTest CMake module). Also, now builds on CI are using CMake, too. Latest commit passed the tests successfully. https://github.com/cqjjjzr/Aegisub/

Next I'll work on writing package scripts using CPack. I think it will be the time for a Pull Request. What's your opinion?