Refactor this project to be more declarative

kminehart commented 1 year ago

"Declarative"?

A declarative model allows users to define the end state rather than the process to reach that end state.

In a way, we are being somewhat declarative: to build grafana, and enterprise together, you supply both flags, which is declarative.

Problem

Some pipelines that use packages as input require running a separate command.
- For example, in order to run the deb command, one would need to run the package command first.
The number of flags is getting quite large. It would be valuable to be able to have a file to inspect and modify that describes the end state.
Each individual command reuses many flags.
Some shared containers have to be repeated when we could have a large DAG where the results of those containers could be reused.
Things like --dry-run can't be easily accomplished.

Solution

This problem could be solved by having a declarative API to define artifacts and how they're built.

Instead of defining a series of commands that do something, there should be one entrypoint, with only flags define the artifacts, and then modifiers that universally affect the entire pipeline.

Requirements

This project should be able to define in Go a list of artifacts, their dependencies (other artifacts), and the steps to produce the artifact.
Their dependencies should be able to be provided via an argument. If they are not provided, then the pipeline should be able to build them.
Users should be able to specify a list of artifacts they want to build.
Users should be able to include a --dry-run flag to view a list of artifacts that will be built and where they will end up.

zerok commented 1 year ago

You mean basically something like go run ./cmd produce file://$PWD/output.deb which then determines based on the output destination what needs to be done? In this case:

Build a grafana tar for the default platform of the user
Generate a deb file out of the tar

?

kminehart commented 1 year ago

I'm thinking that somewhere in the application we have a list of artifacts and the pipelines for how they're produced.

A simple example / just sort of thinking out loud here.

package artifacts

type Artifact struct {
    Dependencies []Artifact
    // ... and other stuff, like a func that returns a dagger.File or dagger.Directory?
}

var Binaries = Artifact{...}
var Frontend = Artifact{...}
var Tarball = Artifact{Dependencies: []Artifact{Binaries,Frontend}}
var Deb = Artifact{
        Dependencies: []string{Tarball},
    }
var Artifacts = []Artifact{
    Binaries, 
    Deb
    Tarball,
    Docker ...,
    RPM...,
}

go run ./cmd --destination=file://dist --deb --docker --rpm

or

go ruin ./cmd --destination=file://dist --deb --docker --dry-run

zerok commented 1 year ago

Okidoki, will start working on this.

zerok commented 1 year ago

Pausing work on the PR for now until the current implementation has been verified. I'll then update the artifact-based implementation.

kminehart commented 1 year ago

I've been thinking about this a little more lately and this could also really help with the containers/ and packages/ file getting huge and complicated.

I think we should start with what we want the interface to look like and then we work backwards.

The goals of the interface are:

Design:

It should be intuitive.
It should have sensible defaults.
It should allow for quick and easy modification, but also
It should allow for easy to digest diffs in git.

Other CLI requirements:

It should allow us to "plan" the results, as in print all of the artifacts and where they'll be placed.
It should allow
It should avoid repeated work. If I ask for a deb:amd64 and I've also requested a targz:amd64 then it should

grafana-build \
 -a deb:arm64 \
-a targz:arm64 \
-a targz:amd64 \
-a targz:plan9 \
--go-image=golang:1.21-alpine \
--destination=gs://grafana-prerelease

OR

grafana-build -i build.json \
--destination=gs://grafana-prerelease

{
    "artifacts": [
        "targz:amd64",
        "targz:arm64",
        "targz:plan9",
        "deb:arm64"
    ],
    "goImage": "golang:1.21-alpine"
}

As for the code, I think we can make things pretty isolated if we categorize them by artifact, like so:

type (
    ContainerFunc func(d *dagger.Client, platform dagger.Platform, args []Argument) *dagger.Container
    BuildFunc     func(ctx context.Context, d *dagger.Client, c *dagger.Container, args []Argument) (*dagger.File, error)
    PublishFunc   func(ctx context.Context, d *dagger.Client, c *dagger.Container, args []Argument) error
)

type Artifact struct {
    Name      string
    Requires  []*Artifact
    Arguments []Argument

    Builder     ContainerFunc
    BuildFunc   BuildFunc
    Publisher   ContainerFunc
    PublishFunc PublishFunc
}

This will give us lots of opportunities to make more reusable functions / containers and categorize dagger code around the artifact that uses it.

guicaulada commented 1 year ago

Could we have -i AND -a? I'd like to be able to create a config file such as the build.json described, but I would also like to run the commands declaratively for a one-off local build.

kminehart commented 1 year ago

yeah I'm thinking we can do a hierarchy of values. Where the argument is available: Environment variables > CLI flags > config file

summerwollin commented 9 months ago

@kminehart I assume the status on this is that it was never started is that correct?

kminehart commented 9 months ago

Completed here but could always be expanded further to support publishing too.

grafana / grafana-build