make clean leaves some dirt.

[FredericHeem]

After pulling new code, invoking make clean and compiling a new version, make now fails with the following message:

make[3]: *** No rule to make target `history/CatchupStateMachine.cpp', needed by `history/CatchupStateMachine.o'.  Stop.

That's already the second occurrence of this issue and the workaround is to clone a fresh repo. The automake/autoconf/configure build tools are archaic, overcomplicated and buggy. I've hated them for too many years and consider them as the world's worst software. Switching to CMake would be a wise move.

[graydon]

Recently we changed to generating input file lists from git's index, rather than manually maintaining them. This in turn means that if you have a stateful source directory tracking git, you periodically need to run ./make-mks to regenerate the input file list from git. I can try to automate this step if you're running into it often, but for the time being just running it manually should solve the problem.

I understand you dislike autotools, but it is unhelpful to couple bug reports with hyperbole about the "world's worst software" or demands to replace the build system we're using. What we have works and is broadly compatible and comprehensible, periodic bugs notwithstanding.

[MonsieurNicolas]

I'm going to close this issue. I really don't care about build systems as long as they work. If the CMake way was easier to maintain and works as well as automake for stellar-core (without requiring dependent projects to switch), I would not be opposed to evaluate a PR - but to get this going probably requires too much effort for little (or no) gain.

[hongxuchen]

I also dislike autotools since too many hacks/tricks are needed in order to make it work, meanwhile it is tightly coupled with GNU toolchains. This is not a big deal for developers of SDF.

The problem I think is that the applied build system is likely hostile to community contributors. In my experience, I'm mostly on Linux/MacOS and I have been using Emacs/Vim for five or six years, however I have never been proficient in C/C++ and keep making mistakes without using IDEs like Clion, Clang/gcc options, or sanity checkers, etc. Every time I would like to digger into Stellar-core implementation, I get lost due to the complexity of the codebase and lacking of navigation support, embedded documentations (compatible with doxygen etc). Indeed I know some tricks to mitigate the situation such as using LLVM/Clang based tools (build capturer, indexer, autocompler, etc), the process is not smooth. Meanwhile, I see other projects such as ethereum/EOS have adopted to CMake system and seem to have embraced more community developers, some of them such as ethereum/Decred also have other implementations rather than C++ due to the relatively more modern toolchains (moreover, even LLVM codebase has deprecated autotools despite that there were quite a lot of legacy issues).

So that's my opinion: in order to attract community contributors, it's better to apply a modern build system like CMake soon. The efforts are worthy in the long run.

And if you are not interested in this adoption, I'd like to have a try to migrate. The challenges I think are mostly the custom commands, the library dependencies, and finally the merging issues. For now, I hope to know what can be thought as an acceptable CMake adoption compared to the current autotools system.

[hongxuchen]

I made some preliminary attempts to migrate to cmake (https://github.com/HongxuChen/stellar-core/commit/bf7c83b406cf82e21fec7de6b4a8d780b7a79c6c, presently only for lib3rdparty, libsoci, libmedida). I'm still trying to figure out how to build xdrpp, libsodium dependencies; the Windows solutions (#1411) are out of my scope for now.

[MonsieurNicolas]

I originally closed this specific issue because it is a "vim vs emacs" type of issue as we already have a build system now that works.

Reason I opened #1411 is that it's a tangible reason to add CMake support to the tree as it would replace manual management of vsproj files (something that we have to deal with at SDF - I am one of those people using Visual Studio as dev environment).

I am not sure I understand how switching build system will help with browsing source code, that seems to be more of an IDE choice, and we do not want to force people in any particular direction here...

In any case, if you have a tangible reason to add support to CMake, create a dedicated issue to discuss it - that said, remember that statements such as "autotools is hostile to the dev community" seems like one of those hyperboles that I am not interested in : if this was the case many open source projects (including Bitcoin core or small projects like Linux kernel) would not have many contributors. So stay on point.

[hongxuchen]

@MonsieurNicolas I think CMake has the following advantages over autotools.

• CMake has nicer cross-platform supports for C/C++ projects. Many autotools' tools are merely bash scripts together with M4 macros, therefore are limited to *nix systems. CMake has a (poorly designed) DSL, and many bundled modules. It works for almost all the generators for all platforms if CMake files have been written well. An significant advantage is that CMake has generators for different Visual Studio versions, MSYS makefile, Unix makefile, ninja etc.
• CMake has more modern features suited for IDEs. Clion is only one aspect. The most important thing I think is the compilation database which can be parsed by different tools like rtags (Emacs, Vim, ST3, atom, VSC, etc), YouCompleteMe (Vim). With autotools, you have to wrap with Bear, which however may fail on partial compilation, and is not portable.
• CMake is easier. The full autotools chains include autoscan, aclocal, autoheader, libtool, GNU make, etc. Despite that some can be wrapped with tools like autoreconf etc, the flow has been complicated. CMake has targets and subprojects. A CMake controlled project structure is comparatively easier to understand, at least there is no need to consider about things like configure.ac, Makefile.am etc. CMake has typically the once-for-all solutions, however figuring out what's going wrong in autotools usually requires understanding of lots of hacking scripts.
• CMake has better (although not excellent) external dependency supports. There have been a few third-party libraries in stellar-core. Some are maintained with git submodules, others are copied with licencing. The problem is still about source code evolution since some old dependency code may have defects (performance issues, vulnerabilities, etc). I feel that with CMake there could be a better solution (but still far from perfect).

Autotools is of course still an option. My concern is that in order to attract community contributors, it's better to introduce more choices (CMake is my favorite, but others like Bazel, SCons etc are still fine). Average programmers may not be as skilled as SDF developers, but you can definitely supervise us (I think Rust community has set up a good example). And my idea is to add CMake build while not breaking autotools support.

[MonsieurNicolas]

More choices is not necessary a good thing here: if you think about what happens over time when those other choices are not maintained properly it makes it even more frustrating for people. I think the only viable way for having support for CMake is to have it maintained with the Windows build (which is why I opened the other issue), and have other platforms be supported unofficially (as they have autotools), at least in the short term. With this approach I’ll support your changes.

[hongxuchen]

The issues for Windows using CMake I think come from pkg-config and subprojects; and some Unix specific build settings like AFL fuzzing cannot be solved.

For subproject issues, if there is a need to index code for xdrpp/libsodium, the elegant way is adding CMake support for these projects themselves (additionally CMake scope issues should be taken carefully), the alternative is writing CMake scripts inside stellar-core but not touching subprojects at all. OTOH, if there is no requirement to index these subprojects, CMake custom targets with some batch scripts are enough IMO. But I don't have a Windows at hand I cannot test them myself.