search

tinyrange / pkg2

Package Metadata Database
https://pkg2.thrush-map.ts.net/
Apache License 2.0
2 stars 0 forks source link

Generate working containers from Neurodocker recipes #10

Open Vbitz opened 5 months ago

Vbitz commented 5 months ago

Neurodocker (https://github.com/ReproNim/neurodocker) is a Python-based tool for easily generating Singularity and Docker container definitions from shell scripts. Pkg2 has had support for parsing these scripts for a while and supports extracting dependency information. With the recent addition of code to export docker containers I'd like to test generating a working container from these "definitions".

I'll start with the recipes defined in https://github.com/NeuroDesk/neurocontainers

Vbitz commented 5 months ago

This depends on #8. The previous version worked but it relied on some shell syntax being stubbed which is no longer the case.

This commit updates the script to the latest version https://github.com/tinyrange/pkg2/commit/e7f7daf27c8974c688599c5d781515c2eded1a71

Vbitz commented 5 months ago

I've got the relevant data extracted from the builder but there are two problems.

Vbitz commented 5 months ago

The other issue is I'm resolving i686 packages when the install was requested for x86_64. I need to propagate the architecture into names unless a override is set for some reason.

Vbitz commented 5 months ago

https://github.com/tinyrange/pkg2/commit/d48979434e4fe20287d04dc6dbec43fc384e7de4 Fixes that issue among others. I've now got a installation plan for afni.

Vbitz commented 5 months ago

https://github.com/tinyrange/pkg2/commit/ce119843fbbe8fb489ed8e7025cf4ae798ee95e6 fetches the contents of the RPM archives. Looking at the basic info it seems like it will be similar code to the Alpine docker image generator (It uses pre and post install scripts, I haven't seen any mentions of triggers).

Vbitz commented 5 months ago

https://github.com/tinyrange/pkg2/commit/351f280ea8c2ae31c4c34909da5f5489a5f737b3 makes a image using the same rough logic as the alpine code. It should theoretically work but sadly the image it generates has too many layers.

It turns out that at least Podman and Docker are limited to 127 layers in a single image (is this a kernel limit?).

I guess I need to write some logic to merge all the layers into a single final layer. That should be easy enough.

Vbitz commented 5 months ago

https://github.com/tinyrange/pkg2/commit/598f50fa0d80d1d40996d33007d80ca8a1dea7c0 merges all the layers together. They are now loadable in Docker.

Now for the installation scripts. RPM supports running scripts with Lua (https://rpm-software-management.github.io/rpm/manual/lua). A bunch of the scripts are written in Lua so I have to figure out how to execute them (and figure how to evaluate them in Starlark later).

I don't seem to have the metadata that indicates if a script is a shell script or a lua script. Let's get that from the metadata.

Vbitz commented 5 months ago

Other projects (like rpm-ostree) have run into the same problem from the looks of things.

The lua code is run by a interpreter embedded inside rpm. More modern versions of rpm offer a tool to run scripts called rpmlua but that was introduced Fedora 37 (https://docs.fedoraproject.org/cs/fedora/f37/release-notes/sysadmin/System_Utilities/).

I guess I have to figure out how rpm runs it's lua scripts to see if there's an alternative entry point. I feel like some of these scripts were introduced pretty recently so the version in Fedora 35 may not be that modern (The AFNI container uses Fedora 35).

Vbitz commented 5 months ago

The functionality is very old https://blog.labix.org/2004/03/23/embedding-lua-interpreter-into-rpm

A quick browse through the code suggests that the scripting functionality only has a few entry points and none that are externally visible.

As far as a way forward is concerned...

Vbitz commented 5 months ago

Turns out nix ran into the same multi-layer issue and came up with a great solution. https://grahamc.com/blog/nix-and-layered-docker-images/ I'll implement a version of the same thing by exposing the installation plan graph to Starlark.

I looked into the RPM Lua issue a little more and it seems only a few initial packages rely on it. I'll need to give more thought to what a solution looks like. rpm-ostree solves the problem by stubbing out the scripts with replacement shell scripts. I could probably do the same thing since I now know it's not that common.

I'm seeing a few smaller file system issues so I'll fix the tar conversion process first. Right now I'm using a hacky little Starlark function to populate the filesystem but I'll replace it with a real converter that takes a StarArchive (Eventually I'd like to replace that API as well but that's a story for another issue).

Vbitz commented 5 months ago

https://github.com/tinyrange/pkg2/commit/7f666aa37f495047f09728d63cca1b30d5a8532f removes make_fs and comments out the RPM scripts for now. They are still in the image but they won't execute when /.pkg/install.sh is called.

Next step is fixing the order of the NeuroDocker directives so I can start running shell scripts as well. I'll implement these as layers which get run in the /.pkg/install.sh script.

Vbitz commented 5 months ago

https://github.com/tinyrange/pkg2/commit/f589a4c609bf30cea3783f30487724a1556c9b6d fixes the order of the directives.

I still need to implement shell script handling. I want to test a bigger variety of the neurocontainer packages besides AFNI first.