An experimental package meta-manager primarily focused on compiled languages, in particular C++.
Currently integrates pacman (both official and unofficial repositories), AUR and Conan.
Takes a propositional logic formula and produces a directory with a set of packages that satisfy the formula. The resulting directory is intended to be used as a root directory of a container image.
Promotes building packages from source with binary caching. Provides means to communicate ABI compatibility to take full advantage of caching.
Supports and generalizes Conan generators for package consumption.
This example assumes you have XDG_RUNTIME_DIR set and that you have Podman installed.
It uses the Docker Compose specification and is compatible with both docker-compose and podman-compose. podman compose is able to run both, see the documentation for more info.
WARNING: docker-compose version 1.X is not supported. Only version 2.X is.
Note that the update will take a few minutes. Updating fetches newly released packages. Do not update unnecessarily, i. e. when you don't want to synchronize with repositories.
Running for the first time after an update also takes a bit longer. Subsequent runs are much faster.
git clone https://github.com/papundekel/PPpackage
cd PPpackage/
podman system service --time=0 &
./examples/update/containerized/update.sh podman
./examples/metamanager/containerized/run.sh podman <examples/input/iana-etc.jsonThe application is highly modular. It is divided into the meta-manager part and multiple different modules. The meta-manager is a driver which communicates with the modules. The modules implement the actual package managers' behavior.
The different types of modules are:
Each module type handles a certain part of the package management process. There are typically multiple modules of each type, each implementing a part of some manager's functionality.
All modules ought to implement a Python interface.
AUR uses the archlinux package utility stack, so except for the logic of the repository, it uses the pacman modules.
The task of the meta-manager is to accept input from the user, parse it into requests which can be forwarded and answered by the modules, and to combine and process the results and present them to the user.
Meta-manager also handles the SAT solving part.
Detailed information about the input format can be found in the Meta-manager Input section. Information about the configuration file is found in Meta-manager Configuration.
The meta-manager works in these basic phases:
A repository driver is responsible for providing information about packages and their relationships.
Some repository information is valid indefinitely, such as dependencies of a certain package version. Others can change in time, e. g. the set of all packages, therefore the driver must provide a current epoch identifier. When the epoch changes, all epoch-bound data are invalidated.
Repository drivers provide epoch as a separate interface and also in each interface providing epoch-bound data. This way the data can be validated against each other so every piece of information is from the same epoch.
The formula is the main piece of data that a repository provides. It describes valid package subsets that can coexist in an installation, i.e. it mainly provides the dependencies. It is a propositional logic formula and its variables are JSON objects that are eventually translated into strings.
The final formula is a conjunction of all repositories' formulas.
In order for repositories to be able to provide packages that interact with each other, some data from repositories needs to be combined into a single structure. This is achieved by the translator data. It is a mapping of symbols to groups which is created from all repositories' translator data.
Its purpose is to provide information about which versions correspond to a single package or a package "provide" in pacman.
The model satisfying a formula is a set of strings. From this set the package manager needs to derive a dependency graph. For this purpose, the repository driver provides a package detail for each package. If a string doesn't correspond to a package, the driver returns a null value.
A package's detail also contains its interfaces and interface dependencies. These are used to construct the dependency graph. Note that in a typical package manager a single dependency mechanism handles both the valid package sets and the dependency graph. In this application, these two issues are separated.
Information which describes how to fetch a package is here called the build context. It can either be:
In the future it would also be possible to support building in a container specified by the image tag or a Dockerfile.
In order to support good caching, package authors and maintainers can provide information for each product. This "product info" is used to 1) identify the product and 2) provide ABI information to depending packages.
The repository driver is responsible for computing a product's info from just the product infos of its dependencies and some information about the build context without building the package. This is essential for caching, as we can determine whether a build would produce an equivalent product without actually building the package.
The requirement translators are mainly responsible for implementing version semantics. Their input is the whole translator info and a requirement from the formula. The output is again a formula, but with variables being strings.
The translators can also provide assumptions, which are variable polarity assignments that the meta-manager is supposed to try to satisfy. This is useful for preferential model selection i.e. when combing pacman and AUR where the user expects AUR packages to be used only if necessary.
Since package installation is quite complex in both pacman and Conan, separate installer modules are used.
Pacman supports running arbitrary hooks while installing a package and also maintains a database of installed packages for updates.
Conan installs all packages into a single cache from which they can be used in builds using generators.
Generators are used by Conan to provide a way to consume packages. Generators provide paths to libraries, CMake files and so on.
The input is taken by stdin and it is in JSON format.
{
"requirements": [
{
"translator": "pacman",
"value": "official-package"
},
{
"translator": "pacman",
"value": "AUR-package=v1"
},
{
"translator": "pacman",
"value": "provide-package>=v2"
},
{
"translator": "conan",
"value": {
"package": "package1",
"version": "1.2.3"
}
},
{
"translator": "conan",
"value": {
"package": "package2",
"version": "[>=2.3.4]"
}
}
],
"generators": [
"conan-CMakeDeps",
"conan-CMakeToolchain"
]
}The requirements are currently a conjunction but could be any propositional formula.
The translator field specifies the requirement translator to be used for translating that particular requirement. The value field is the actual value of the requirement.
pacman translatorAny string valid in pacman depends or conflicts field. This means a (virtual) package name with an optional version specification.
conan translatorDictionary with package name and version requirement.
{
"package": "package",
"version": "version"
},Corresponds to package/version in Conan.
The configuration is a JSON file and is passed to the meta-manager by the --config CLI option.
{
"cache_path": "/path/to/cache/",
"containerizer": {
"url": "unix:///path/to/podman/or/docker.sock"
},
"repository_drivers": {
"driver1": {
"package": "python.package.driver1",
"parameters": {
"implementation_defined": ""
}
},
"driver2": {
"package": "python.package.driver2",
"parameters": {
"implementation_defined": ""
}
}
},
"translators": {
"translator1": {
"package": "python.package.translator1",
"parameters": {
"implementation_defined": ""
}
},
"translator2": {
"package": "python.package.translator2",
"parameters": {
"implementation_defined": ""
}
}
},
"installers": {
"installer1": {
"package": "python.package.installer1",
"parameters": {
"implementation_defined": ""
}
},
"installer2": {
"package": "python.package.installer2",
"parameters": {
"implementation_defined": ""
}
}
},
"generators": {
"exact": {
"package": "python.package.generator1",
"parameters": {
"implementation_defined": ""
}
},
"prefix*": {
"package": "python.package.generator2",
"parameters": {
"implementation_defined": ""
}
}
},
"repositories": [
{
"driver": "driver1",
"parameters": {
"implementation_defined": ""
}
},
{
"driver": "driver1",
"parameters": {
"implementation_defined": ""
}
},
{
"driver": "driver2",
"parameters": {
"implementation_defined": ""
}
}
]
}
The meta-manager is able to generate a dot file with the resolution graph.
python -m PPpackage.metamanager --graph <graph_path> ...For all testing scenarios, a clone of the repository is required.
All scripts expect to be run from the git repository root.
They create directory tmp/ to store all files created during the run of the program.
git clone https://github.com/papundekel/PPpackage
cd PPpackage/The installation directory will be locate in tmp/root/. Generators and the resolution graph are located in the tmp/output/ directory.
Note that the program uses caching and so the first run is very slow compared to subsequent ones.
All scripts are located in the examples/ directory.
The manager can be invoked natively or in a container.
The meta-manager launches containers and needs access to a containerizer. Currently only Podman is supported, but the design of the project allows to easily support Docker as well.
All examples use config unix://$XDG_RUNTIME_DIR/podman/podman.sock for the containerizer socket url which is the default location for Podman. You can change this value to any location in the config.json files according to your setup.
The SAT solver the meta-manager uses is invoked as a container. Therefore the image for the solver needs to be built.
./image-build.sh $containerizer solverIt is possible to test the application directly on the host machine without any containerization.
As all applications in these scenarios run on the host, we need to install the required packages first.
python -m venv .venv/
source .venv/bin/activate
pip install --requirement requirements-dev.txtpyalpm, a Python bindings library for libalpm, requires libalpm to be installed manually. The pacman installer uses fakealpm which has dependencies.
archlinux:
pacman -Syu libalpm gcc cmake ninja boost nlohmann-jsonUbuntu:
apt install libalpm-dev gcc cmake ninja-build libboost-dev nlohmann-json3-devTo build fakealpm:
./installer/pacman/fakealpm/build.sh installer/pacman/fakealpm/ installer/pacman/fakealpm/build/ $fakealpm_install_dirIf fakealpm_install_dir is set to something else than /usr/local/ then you need to configure fakealpm_install_path parameter in the pacman installer to that path.
It is also possible to run the application using the Compose Specification. Both Docker and podman are supported.
The only requirements are therefore Docker or podman and a composer (docker-compose or podman-compose).
WARNING: docker-compose 1.X is not compatible. Use version 2.X.
It is optional to build the images yourself. They are hosted on dockerhub.
./image-build.sh $containerizer metamanager
./image-build.sh $containerizer updaterDocker requires more configuration because of how
user namespace mappings work, so the compose files are written to work for podman.
Support for Docker can be added to the compose file by supplying the USER environment variable to the composer and Dockerfile and bind mounting the /etc/passwd
and /etc/group files. An example of this configuration can be seen in the github workflow in .github/compose.yaml.
Before using the manager, databases of used repositories need to be created. The examples contained in this repository always use the same configuration of five repositories. To create or update their databases, use one of the following methods.
./examples/update/native/update.sh./examples/update/containerized/update.sh $containerizerThese scripts create files in ~/.PPpackage. It is also the location where run scripts look for the files. All locations are configurable but are left to their simplest defaults for ease of testing.
./examples/metamanager/native/run.sh <examples/input/iana-etc.json./examples/metamanager/containerized/run.sh $containerizer <examples/input/iana-etc.jsonThere are multiple input examples in the examples/input/ directory, you can try any of them.
One example project is also provided. It is the project described in the Conan documentation. It resides in examples/project/compressor/.
First, the build context for the project is created with our meta-manager:
./examples/metamanager/$method/run.sh < examples/project/compressor/requirements.jsonThis creates the image rootfs in the tmp/root directory and generators in tmp/output/generators. Then we can run the provided script, which builds the project with build script examples/project/compressor/build.sh. The script is just the modified version of commands run in the Conan documentation. The paths to the image rootfs and generators need to be absolute.
./examples/project/compressor/build-in-container.sh $containerizer $PWD/tmp/root $PWD/tmp/output/generators
./examples/project/compressor/build/compressorWe can also invoke the meta-manager directly without the run.sh scripts and then we would not have to move the directories as we could specify the output directories directly. The only problem with this method is that the containerized meta-manager needs to have path translations for the containerizer set for the root directory and that requires changing the config.json file. The native version doesn't have this problem as it resides in the same mount namespace as the containerizer.