Proper project build configuration

vsbogd commented 1 year ago

At the moment the project has two CMake projects inside:

hyperonc which builds C API library
hyperonpy which builds Python API library using C API one

hyperonpy depends on hyperonc and they can be built one by one. As hyperonpy links hyperonc statically the developer need to rebuild both hyperonc and hyperonpy when C API is changed. To simplify development a root CMake projects is added. hyperonc and hyperonpy are included into the root project as external projects in order to allow them be built by a single cmake/make command.

It turns out this configuration doesn't work properly. In particular it doesn't propagate the CMAKE_BUILD_TYPE from root project to hyperonc and hyperonpy subprojects (see https://github.com/trueagi-io/hyperon-experimental/pull/377 which is quickfix). It also doesn't allow using make test target on the root project to test subprojects (this is workarounded by introducing custom make check target).

This issue is to work on proper project configuration. There are number of things to try:

[ ] use CMake FetchContent instead of ExternalProjects
[ ] include hyperonc project into hyperonpy project and remove root project
[ ] replace Python CMake project by PYO3 (see https://github.com/trueagi-io/hyperon-experimental/issues/283) and may be it changes the layout of the project.

vsbogd commented 7 months ago

This comment is to explain the use-cases of the build system we are supporting, explain which difficulties exist and discuss what can be done to make a project setup more easier.

The chain of the compilation dependencies we have

                      libcheck    pybind11 + option-lite
                         v                 v
metta-lib (./lib) --> metta-c (./c) -> metta-py (./python including hyperonpy.cpp)
                  \-> metta-repl (./repl)

Use-cases to support.

1. Standard dev mode

Developer clones repository and changes the code. Need a single command to build or test the whole chain of native projects after changing any of them. Changes in Python files should be applied without rebuild.

2. Release packages

Each component is released and published independently, next component is released using release version of previous one. Release version is received from artifacts storage. Cargo workspace publishing should do it automatically. metta-py component is released building metta-c version from crates.io.

3. Build from source and use.

User gets a copy of repo to use the latest version of binaries but without needness to modify the source code frequently. In such case the common way is to build binaries from source, install them into system using make install or similar and use them.

Rust components can be build easily using Cargo but there is no Cargo command to install metta-c into a LFS to allow metta-py finding it. metta-py should have separate project to compile hyperonpy.cpp into a binary Python module.

vsbogd commented 7 months ago

Current state of the build system

Each component can be built separately:

pure Rust components (metta-lib and metta-repl) has Cargo project file.
Rust/C component (metta-c) has both Cargo and CMake project files. CMake project file is used to implement make install which is absent in Cargo for libraries and export dependency to the other CMake projects.
Python component (metta-py) has CMake setup to build a native part (hyperonpy.cpp) and Python project setup (setup.py + pyproject.toml). setup.py implements two cases:
- Copy pre-build hyperonpy library from ./python directory if it is present. Now I am not sure in which case it is triggered. In dev mode built library is already present in ./python directory. In other modes it should be built.
- Build hyperonpy library from source distribution. It is expected that user already installed hyperonc library properly thus CMake can find it by using standard paths. This approach is used to release binaries or install Python component separately. When Python library is released inside cibuildwheel docker then install-hyperonc.sh script is used to install hyperonc from sources.

CMake is used to orchestrate the build in a dev mode: it builds metta-c and metta-py and ensures projects see each other.

What is good in current state

Rust projects can be built by one command which is handy for pure Rust devs who wants to play with code. Central CMake script allows building and testing the C and Python related part by a single command. Python module can be installed in editable mode and all changes in Python files are automatically applied. Changes in Rust part are applied after rebuild. All native dependencies are automatically downloaded.

Known issues

metta-c library has two different build scripts: Cargo to build Rust related code, CMake to run C unit tests, install C libraries. make install usually requires root privileges but Rust cannot build the project under the root user (even building a project using make and then use make install doesn't work). Thus one should use install path which doesn't require root privileges. On the other hand Python project cannot be configured to use custom root prefix, thus use-case 3 is not implemented.
Orchestrating CMake script uses metta-py CMake script directly hijacking its behaviour via variables. Second step is needed to install Python project in a dev mode. One cannot use just pip install -e ./python either.
Conan version 1.xx we use to get dependencies is not longer developed, there is some gap between new compilers released and they are supported by Conan (see https://github.com/trueagi-io/hyperon-experimental/issues/421).
Overall solution looks a bit complex: Cargo, CMake, Conan.

trueagi-io / hyperon-experimental