Open skant7 opened 2 years ago
I am going to detail here all the images with dependencies, so we define a standard way to name them and also the meaning of each one. I am going to do this in Python.
First of all, we have the base image which is debian-slim
, the version can be configurable but right now we are using debian:bullseye-slim
.
Then we have the dev
kind of image. This kind of images are used for developing the core itself (i.e building it):
dev-deps-base
: It includes dependencies like gcc, git, ... (https://github.com/metacall/core/blob/77f5a0e74d304ac69578f43bf47ede55e9a3063e/tools/metacall-environment.sh#L62), and also the cloned repo of MetaCall. It is the base of all dev
images, needed for building any loader.dev-deps-py
: This includes the dependencies for building py_loader.so (https://github.com/metacall/core/blob/77f5a0e74d304ac69578f43bf47ede55e9a3063e/tools/metacall-environment.sh#L92).dev-py
: This is the previous image but with py_loader.so built.Those images inherit one from each other, and we have the last two repeated for each language (they are independent between languages).
Another kind of images, which are independent to the compilation of metacall/core itself are the runner
s. Those do not need development dependencies (in terms of metacall/core development) but they need end user development dependencies. For example, in Python, they will need python3
and pip3
in order to install dependencies and run python code, but it won't need python3-dev
package, instead it will be enough with python3-lib
which is already included by python3
. This is needed in order to have a minimal image in which to run installation scripts (ci/cd) of the end user projects. They do not need to inherit from dev-deps-base
(usually). There's some exceptions, like for example, sometimes NodeJS has requirements like electron which need a compiler in order to install them, similar can happen in python although I haven't seen that in the same frequency.
Those images will be used as intermediate images in order to build the final image of the end user. They will execute pip install ...
or npm install ...
during "end-user build time" (i.e during when we are creating the end user images, not the images of metacall/builder itself).
Once we have all those images, we need the runtime images. Runtime images are for executing code based on metacall and they are based on debian-slim
. For this we need the following (Python):
1) Runtime dependencies of Python: https://github.com/metacall/core/blob/77f5a0e74d304ac69578f43bf47ede55e9a3063e/tools/metacall-runtime.sh#L62
2) MetaCall runtime dependencies: py_loader.so
The 2) point can be taken from this operation: diff(dev-deps-py, dev-py)
and excluding the build folder, because it will have .o
files related to the compilation step.
The dev images are not really important for the end user, maybe we can find some use case for metacall/core development but in my opinion we can leave them for now... for development we can use the tools in metacall/core and compile it with multiple languages at once.
I am trying to write the build phase in docker syntax but it is not possible due to the limitations, maybe I can draw some graph instead. But basically we must have runtime images with only the dependencies for runtime, i.e a debian-slim
with py_loader.so + libpython3. For each language merge then and copy the end user dependencies. This can be the first approach, avoiding runners. Once we have it, we can add runners by adding an intermediate phase where runners are executed, and the dependencies of the user are copied from the runners instead.
This design looks good to me, however I have few questions :
dev-deps-base
image needs to be built initially or does it gets built during building the loaders phase.pip
to mock the runners ?libpy_loader.so
+ libpython
) ?Buildkit in deamonless+rootless mode (for testing):
docker run \
-it \
--rm \
--security-opt seccomp=unconfined \
--security-opt apparmor=unconfined \
-e BUILDKITD_FLAGS=--oci-worker-no-process-sandbox \
-v /path/to/dir:/tmp/work \
--entrypoint buildctl-daemonless.sh \
moby/buildkit:master-rootless \
build \
--frontend \
dockerfile.v0 \
--local context=/tmp/work \
--local dockerfile=/tmp/work
Check out how Moby creates his images: https://github.com/moby/buildkit/blob/354e4f6e6b2008a6690b1675279e6bd2e445d492/Dockerfile#L86
The base image is debian-slim. (You can specify the base image using the --image flag) The other images currently supported are of dev, deps and runtime images. The specifics are mentioned in the above comment here
Here py is an example for python, you can pass multiple args to builder py c etc. as languages for building required image.
Deps images :
Dev image :
Taking diff(deps base, deps base py)
will give us the dev dependencies for python.
and taking diff(runtime base, runtime py)
will give us the runtime dependencies for python.
So, to form the final runtime image containing the python loaders as well as the python runtime.
We do merge(runtime base py, diff(dev base, dev base py)
- the second part here i,e diff(dev py, dev base) are taken after removing the build folder in each to avoid conflicts in diffs if any. It then finally will basically give you the loaders for python.
Here we are going to define the design of builder
Goals: