Open domenkozar opened 1 month ago
That looks useful. Would it also be intended for devenv inner workings? For instance when requirements.txt changes, run pip install in 'enterShell' when languages.python.venv.requirements
is set?
That looks useful. Would it also be intended for devenv inner workings? For instance when requirements.txt changes, run pip install in 'enterShell' when
languages.python.venv.requirements
is set?
Yeah exactly, I hope that a lot of the "state changed, let's do work" can be captured under this abstraction so that we can do better logging and composability.
One possible design sketch with more explicit naming (I've never really liked 'task' for this):
{ ... }: {
converge.constraints.add "bundler" {
impl = "bundle-impl.rb";
depends = [];
};
converge.constraints.add "db-migrations" {
satisfied = "bin/check-db";
satisfy = "bundle exec rake db:migrate";
depends = [ converge.constraints.bundler ];
};
converge.goal = [ converge.constraints.db-migrations ];
}
This could generate some file like...
# converge.json
{
"default": ["db-migrations"],
"constraints": {
"bundler": {
"impl": "bundle-impl.rb",
},
"db-migrations": {
"satisfied": "bin/check-db",
"satisfy": "bundle exec rake db:migrate",
"depends": ["bundler"],
}
}
}
And we could have a tool invoked like converge --file converge.json db-migrations
I earlier mentioned a case I was stuck on where i wanted to make changes to the guest user and password and other configurations in rabbitmq that's not available as config arguments but have to be set via the command line, but I couldn't find a way of doing this within the same rabbitmq process or calling a script within devenv.sh that runs for that particular service alone, in this case rabbitmq Does this also address such usecases?
Hmm, I hadn't used Just before, but it seems it cannot handle file dependencies. See https://github.com/casey/just/issues/867. This makes it less useful for cases like running pip when requirements.txt has changed. Make might be more fitting? Or ninja?
met = "sqlx ";
depends = [];
meet = "bundler";
gemfile = "Gemfile";
This mixes the dependency graph structure (depends
) and evaluation (met
, meet
). That seems generally fine in terms of of devenv
. I'm not sure how else that you'd determine "whether" and "what" to do at a particular node in the dependency DAG.
In the ideal case, IMO, it'd be an improvement to understand the DAG as some kind of "program". Both met
and meet
are two related functions that are associated with a symbol (in this case, the name of the "task), and evaluated against an attribute-set of "node data". If we were interpreting in an O-O context, then, perhaps, the symbol would be a class name to be instantiated, the attribute-set it's initial state, and met / meet
are methods on that instance.
If it's not possible to represent this expression as a complete "program", then it's just make
where the ability to correctly interpret the DAG depends on a multitude of other tools that we must invoke.
Perhaps the way out of this would be if the met
and meet
attributes referenced derivations that might be implemented within the derivations provided by devenv
or from external inputs to the devenv.nix
.
Something like:
{ inputs, pkgs, ... }:
{
# rename tasks => targets because that's the traditional name
task.targets = [
# first thing in the array is default
"db-migrations",
"bundle"
]
targets.db-migrations = {
# maybe this repository uses an input that offers a better database check
met = inputs.database.checkDatabase;
depends = ['bundle'];
# let's imagine that languages.ruby provided us with rake
meet = rake.dbMigrate;
};
targets.bundle = {
# again, let's imagine that languages.ruby gives us more derivations to work with
met = ruby.bundle.state;
meet = ruby.bundle.install
};
}
The example exactly addresses a need I have. Nice. I have a process, initial-migration
, that runs every time I run devenv up
and it checks state to see if migrations need to execute and no-ops otherwise. Generally, it’s only useful on the initial environment setup and is just noise in the process-compose processes list after that point.
Aside: I saw a super neat looking progress indicator for tasks in some video covering the latest Zig release:
https://youtu.be/_rcD_V1oPus?t=225 / https://asciinema.org/a/661404
There's a blog post specifically covering the implementation of that here: https://andrewkelley.me/post/zig-new-cli-progress-bar-explained.html he calls it "The Zig Progress Protocol Specification".
It could be worth looking at https://cuelang.org/ to meet the requirements described here
Here is an example
The below is pseudocode example of creating schemas to define tasks in cue, and a generalized runner for the tasks
package main
import (
"encoding/yaml"
"encoding/json"
)
// Load YAML file
yamlData: yaml.Unmarshal(#readFile) & #TaggedFile
#TaggedFile: {
#readFile: string
if #readFile == _|_ {
!!! "Cannot read YAML file"
}
}
// Task schema
#Task: {
name: string
cmd: string | [...string]
deps?: [...string]
check?: [...string]
env?: [string]: string
}
// Validate tasks against schema
tasks: [string]: #Task
tasks: yamlData.tasks
// Add any additional validations or derived fields here
tasks: [string]: {
// Example: Ensure all task names are capitalized
name: =~"^[A-Z]"
}
// Command to output the task graph as JSON
command: output: {
task: json.Marshal(tasks)
stdout: task
}
This is an example of a yaml file cue can consume (could be json or other format too)
tasks:
setup:
name: "Setup Environment"
cmd: ["./setup_script.sh"]
build:
name: "Build Project"
cmd: ["make", "build"]
deps: ["setup"]
test:
name: "Run Tests"
cmd: ["make", "test"]
deps: ["build"]
check: ["test", "-f", "some_file"]
deploy:
name: "Deploy"
cmd: ["./deploy.sh"]
deps: ["test"]
env:
DEPLOY_ENV: "production"
This can be run with a Rust or Go program, output task graph, and run in parallel
(pseudo Go example but could be Rust too of course)
package main
import (
"encoding/json"
"fmt"
"os/exec"
"sync"
)
type Task struct {
Name string `json:"name"`
Cmd []string `json:"cmd"`
Deps []string `json:"deps,omitempty"`
Check []string `json:"check,omitempty"`
Env map[string]string `json:"env,omitempty"`
}
func main() {
// Run CUE to get task graph
cueCmd := exec.Command("cue", "cmd", "-t", "#readFile=./tasks.yaml", "output")
output, err := cueCmd.Output()
if err != nil {
fmt.Println("Error running CUE:", err)
return
}
var tasks map[string]Task
err = json.Unmarshal(output, &tasks)
if err != nil {
fmt.Println("Error parsing JSON:", err)
return
}
// Execute tasks
var wg sync.WaitGroup
for name, task := range tasks {
wg.Add(1)
go func(name string, task Task) {
defer wg.Done()
executeTask(name, task, tasks)
}(name, task)
}
wg.Wait()
}
func executeTask(name string, task Task, allTasks map[string]Task) {
// Wait for dependencies
for _, dep := range task.Deps {
<-taskCompletionChannels[dep]
}
// Execute task
fmt.Printf("Executing task: %s\n", name)
cmd := exec.Command(task.Cmd[0], task.Cmd[1:]...)
cmd.Env = append(cmd.Env, formatEnv(task.Env)...)
err := cmd.Run()
if err != nil {
fmt.Printf("Error executing task %s: %v\n", name, err)
}
// Signal completion
taskCompletionChannels[name] <- true
}
[ ... ]
Of course it could be possible to just cut cue out of the picture, and consume and run tasks directly in Rust too. But cue can do merging of config and some other things so I thought it worth a consideration.
I have quite a similar prototype written in Rust!
I'll try to wrap it up and it's so liberating to see we've arrived at similar results :exploding_head:
Bit of a weird suggestion, but moonrepo/moon is pretty awesome.
Sadly they built their own tool, proto, to handle software acquisition; but they are planning to support other tools. Perhaps we don't need to invent anything here and instead collaborate to support Nix?
Tagging @milesj as he may be able to share some input or thoughts
@rawkode that looks super nice, but I'm concerned that it tries to do too much with things like dependency management.
There's also https://bob.build/, but it suffers from the same issue.
https://taskfile.dev/ looks extremely promising!
https://taskfile.dev/ looks extremely promising!
Was watching this to get a better understanding of how Just can integrate with Nix: https://youtu.be/wQCV0QgIbuk and a comment recommended the above over Just - for whatever that’s worth.
I really don’t like working with/writing YAML; could we use Dhall or something and generate the YAML for Task? 👀
I've dumped my thoughts what we'd need for devenv.sh at https://github.com/go-task/task/issues/448#issuecomment-2288918354
We're using Nix to do congruent configuration where possible, but in non-Nix world we often need to do convergent configuration.
We're using mostly a glue of bash at the moment, in phases like
enterShell
andenterTest
, etc.Problems
Requirements
Syntax
Command
We'll need to find a good tool that we can use under the hood, one such tool is Justfile see #1320