Configuration driven commands

tuler commented 1 month ago

📚 Context

The current CLI commands like build and run currently allow some level of customization through the following mechanisms:

machine rootfs contents from a Docker image
RAM memory size from a Docker label
drive extra space from a Docker label
emulator version (or kernel) by replacing an underlying "sdk Docker image"

This process is kind of scattered and makes it hard to allow further customization required by some applications or different frameworks like lambada.

✔️ Solution

Implement build and execution driven by a configuration file. The configuration will be a TOML file containing sections for machine drives, kernel configuration, runtime configuration, etc.

The configuration file content can be discussed further in this issue. Ideally the file can be considered optional for a project if all options have a default value.

📈 Subtasks

[ ] define the file content, and the default values
[ ] implement the build command using the configuration
[ ] implement the run command using the configuration
[ ] implement the deploy command using the configuration

🎯 Definition of Done

[ ] build based on configuration
[ ] run based on configuration
[ ] allow novel use cases for the CLI, like lambada applications

guidanoli commented 1 month ago

So, right off the bat, I think these labels would turn into key-value pairs in this TOML file, right?

LABEL io.cartesi.sdk_version=0.6.0
LABEL io.cartesi.rollups.ram_size=128Mi
LABEL io.cartesi.rollups.data_size=128Mb

:arrow_right:

sdk_version = "0.6.0"
ram_size = "128Mi"
data_size = "128Mb"

tuler commented 1 month ago

So, right off the bat, I think these labels would turn into key-value pairs in this TOML file, right?

Like that, but with a better structure.

First, for drives I'm considering something like:

[drives.root]
filename = "rootfs.ext2"

Where root is the drive label. Filename is the ext2 or sqfs filename, in case the drive already exists (from another external process).

For the drives that are about to be created by the build command, instead of defining a filename I'd define a builder = docker property, and an optional dockerfile with the path to Dockerfile, like this:

[drives.root]
builder = "docker"
dockerfile = "Dockerfile"
extraSize = "10Mb" # default 0
format = "ext2" # or "sqfs"
# tags = "my-dapp"
# target optional

The above can actually be the default behavior, so It would work as it is now.

In addition there are the following properties, which are self-explanatory.

shared = true
mount = "/"
user = "dapp"

I'm not sure if the drive memory start and size should be configurable. Maybe there is a use case for that.

diegonehab commented 1 month ago

For version 0.20, I want to have the simplified API for the machine in place. The one that reduces the number of methods in the class, and that uses JSON to specify the machine configuration to simplify the types. We might also remove the other complicated structures from the C api, such as the proofs and the log, and use JSON for those as well.

I believe we should make the drive labels part of the machine config. At the moment, labels are a cartesi-machine.lua concept that only exists when you are creating a machine. At that point, cartesi-machine.lua copies this info into /run/drive-label using the init script. But if you load a machine from disk, cartesi-machine.lua has no idea about labels. Code inside the machine still has this info, but whoever is controlling the machine has not.

Labels might be important when we move to lambda. Or when you have overlays etc. Maybe we should add them to the introspection for memory ranges as well. You could programatically iterate over the memory ranges, filter out the drives, and look at the labels to figure out how to treat them. Or go via configs. What do you think?

What other use were you conceiving for the labels?

As for the file structure, I would prefer if you separated the specification of what a drive is in the machine (this is the machine structure) from the specification of how to build a certain image file. Perhaps something like this?

[drives.root]
filename = 'rootfs.ext2'
start = 8 << 52

[drives.lambda]
filename = 'state.raw' # I think we should support raw drives 
shared = true # this would be for all lambda drives?
user = dapp
mount = false
start = 9 << 52

['rootfs.ext2']
builder = docker
dockerfile = Dockerfile

['state.raw']
builder = pristine
length = 10Mi

tuler commented 1 month ago

Perhaps something like this?
['state.raw']
builder = pristine
length = 10Mi

I don't like that format very much. It's not very TOML'ish to have dynamic keys like that, except when "under" a certain key, like what I did with drives.<label>. Usually TOML keys are known and parsed explicitly. The parse of your suggestion would have to follow a dependency mechanism like what happens with Makefile (which is probably where you are coming from).

tuler commented 1 month ago

uses JSON to specify the machine configuration to simplify the types

That feature of the cartesi-machine may have a positive impact on this feature of the CLI, because the CLI could either accept a machine JSON from the application, instead of replicating those in a TOML format, and leave the TOML only for drive building procedure definition.

The other option would be to continue on this path of defining machine configuration in TOML, and generating a JSON for the cartesi-machine, which will most likely be required anyway when 0.20 comes out, unless you keep compatibility with the currently behavior of cartesi-machine.lua.

What are your thoughts on that @diegonehab

cartesi / cli