domenkozar commented 2 weeks ago

Task Server Protocol

https://devenv.sh allows defining tasks in your favorite language using JSON-RPC protocol and exposing them as an executable :

{
  task.serverProtocol = [ "myexecutable" ];
}

Listing tasks

Which would launch myexecutable /tmp/rando.sock and devenv would on startup immediately ask for a list of tasks:

{
  "jsonrpc": "2.0",
  "method": "initialize",
  "params": {},
  "id": 1
}

And the server responds:

{
  "jsonrpc": "2.0",
  "result": {
    "tasks": [{
      "name": "prefix:custom",
      "after": []
    }]
  },
  "id": 1
}

Running tasks

Then the client can ask the server to run a task:

{
  "jsonrpc": "2.0",
  "method": "run",
  "params": {
    "task": "prefix:name",
    "inputs": {},
    "outputs": {}
  },
  "id": 2
}

And server streams updates about stdout/stderr:

{
  "jsonrpc": "2.0",
  "method": "log",
  "params": {
    "task": "prefix:name",
    "line": "some text",
    "stderr": false,
    "time": "20240828T212611.11Z",
  }
}

Until the final response from the server with outputs and the final status of the task.

{
  "jsonrpc": "2.0",
  "result": {
    "task": "prefix:name",
    "outputs": {},
    "status": "succeeded"
  },
  "id": 2
}

Rust SDK

use task_server_sdk::{Task, TaskServerProtocl, cli::Args};
use clap::Parser;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let tasks = vec![
        Task {
            name: "myapp:task1".to_string(),
            after: vec![],
            exec: Box::new(async {
                println!("Executing custom task");
                Ok(())
            }) as Box<dyn Future<Output = anyhow::Result<()>> + Send + Unpin>),
        },
    ];

    let args = Args::parse();
    TaskServerProtocl::run(tasks, &args).await
}

Implementation

Use reth-ipc for the Client using jsonrspee
A test suite with a defined set of tasks that Server SDK should implement for the test suite to run against
Task Server Protocol implementation

bobvanderlinden commented 2 weeks ago

I was a bit surprised by this. I thought the tasks were usually short-lived operations that just run one or more commands in a certain order and then just end.

The TSP seems to indicate this is a long lived process that can interactively start one or more tasks. It almost seems like the protocol could be used for the processes of devenv up?

That's pretty cool, I'm just wondering what the intention is.

Just to illustrate, the same could be done by myexecutable run task < json_file? Where myexecutable always lives as long as the tasks it's running.

That said, the methods/responses make sense to me. I just wasn't sure where the tsp would get its configuration from if that resides in devenv.nix?

Nitpick (feel free to ignore): The stderr being bool was a bit surprising, but the alternatives I thought up might not be better (io: stdout or even fileNo: 1).

domenkozar commented 1 week ago

@bobvanderlinden you're correct to spot the intention for tasks to be an implementation detail for processes. That way we get systemd-level dependency trees also for processes.

Just to illustrate, the same could be done by myexecutable run task < json_file? Where myexecutable always lives as long as the tasks it's running.

There are two issues here when I was thinking about the design:

providing language specific interface via command line parsing is inflexible for future changes
we need a way for myexecutable to report logging, etc at which point you're already introducing bidirectional channel

I just wasn't sure where the tsp would get its configuration from if that resides in devenv.nix?

it gets configuration from initialization phase that happens as soon as executable runs.

Nitpick (feel free to ignore): The stderr being bool was a bit surprising, but the alternatives I thought up might not be better (io: stdout or even fileNo: 1).

I had the same thought, the alternative would be line_stdout and line_stderr?

domenkozar commented 1 week ago

See https://github.com/cachix/devenv/issues/1471

cachix / devenv