search

cue-lang / docs-and-content

A place to discuss, plan, and track documentation on cuelang.org
6 stars 1 forks source link

placeholder: use Go API and a loaded CUE schema to vet yaml file #6

Open myitcv opened 1 year ago

myitcv commented 1 year ago

Based on https://cuelang.slack.com/archives/CLT3ULF6C/p1694699710517009

Thread ``` Hey gang. I’d like to use a CUE file where I’ve defined a schema definition, and load it into a Go program, then unify a struct that I’ve already parsed using go-yaml. I’m having a hard time parsing whether this is a problem for [cuelang.org/go/cue/load](http://cuelang.org/go/cue/load) or [cuelang.org/go/cue/cuecontext](http://cuelang.org/go/cue/cuecontext)? Any tidbits or examples I could use to get started? 43 replies mvdan [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694700360815749?thread_ts=1694699710.517009&cid=CLT3ULF6C) you can only unify cue.Value values, so you'd have to first convert that Go value you parsed via go-yaml into one. I believe you could do that with https://pkg.go.dev/cuelang.org/go/cue#Context.Encode mvdan [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694700399640809?thread_ts=1694699710.517009&cid=CLT3ULF6C) https://pkg.go.dev/cuelang.org/go/cuego is a similar package that is higher level, but note that it is mainly for validating constraints stored in Go struct field tags, so you can't supply your schema as a CUE value mvdan [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694700471220249?thread_ts=1694699710.517009&cid=CLT3ULF6C) once you are done unifying your schema with your CUE-encoded Go value, you can get it back to a Go value with https://pkg.go.dev/cuelang.org/go/cue#Value.Decode, if you wish mvdan [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694700490120229?thread_ts=1694699710.517009&cid=CLT3ULF6C) you can think of the Encode and Decode methods as a conversion layer between cue.Value and Go values, i.e. interface{} jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694710106184399?thread_ts=1694699710.517009&cid=CLT3ULF6C) Is there a way to use a CUE file to generate Go structs so the CUE file can be the source of truth and I don’t need to keep them in synch? jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694710195161859?thread_ts=1694699710.517009&cid=CLT3ULF6C) Seems like [cuelang.org/go/encoding/gocode](http://cuelang.org/go/encoding/gocode) might do that? jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694711509583649?thread_ts=1694699710.517009&cid=CLT3ULF6C) Or is there a way to use go:generate? jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694711703026749?thread_ts=1694699710.517009&cid=CLT3ULF6C) Seems like I’m looking for “TypeGen”. https://cuetorials.com/deep-dives/code-gen/ jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694713983358859?thread_ts=1694699710.517009&cid=CLT3ULF6C) Found this: https://docs.hofstadter.io/code-generation/type-and-structs/ Tony Worm [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694714117051359?thread_ts=1694699710.517009&cid=CLT3ULF6C) I have a certain approach I take with hof primarily, using text/template and things like fields: [n=string]: { name: n type: "string" default: "foo" } This is the base "schema", and then you can enrich these for language or technology specifics, sometimes this requires the end end user fill in some of the added fields. If you look around https://github.com/hofstadter-io/hof/tree/_dev/schema/dm https://github.com/hofstadter-io/hofmod-types (out of date, still same concepts) The main drawback of this approach is that you cannot use a more CUE like schema with incomplete fields and constraints (yet, this could be another enrichment, but better written in Go). However, you cannot pass incomplete CUE to the text/template engine either. So you have to define code gen output via complete values and text/template (in)complete values and Go code string manipulation jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694715115404739?thread_ts=1694699710.517009&cid=CLT3ULF6C) I’m confused. This looks like you’re writing CUE to define fields and types to generate stubs, when I’ve already got a CUE file that has that field and type information in it? jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694715162339399?thread_ts=1694699710.517009&cid=CLT3ULF6C) So you would still have to maintain a CUE file for the data, and a CUE file for stub generation? jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694715249206899?thread_ts=1694699710.517009&cid=CLT3ULF6C) I’m trying to bend my brain, but I’m really struggling. :slightly_smiling_face: jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694715824061509?thread_ts=1694699710.517009&cid=CLT3ULF6C) I guess I was looking for an easy to get the data back out of CUE into a Go struct after it’s been validated, but maybe I’m not thinking about it correctly. myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716561425139?thread_ts=1694699710.517009&cid=CLT3ULF6C) [@jesselang](https://cuelang.slack.com/team/U05QS7GPJDS) - you're actually holding it just right in your head. The problem is that CUE's support for encoding to/decoding from various different data/schema/other languages is incomplete. Specifically, we do not yet support generating Go types from CUE. But it's absolutely something we can and will support jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716626421019?thread_ts=1694699710.517009&cid=CLT3ULF6C) I want to work with the data easily after validation. It can stay in CUE, but that seems tough for Go devs who don't want to have to think about CUE to swallow. myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716633393749?thread_ts=1694699710.517009&cid=CLT3ULF6C) (note that I will leave Tony to discuss hof here - the approach hof takes is different. Now that it is "right" or "wrong" in what it does, rather I want to clarify the context of what the CUE project itself will do in this area). myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716704716949?thread_ts=1694699710.517009&cid=CLT3ULF6C) Fundamentally, you should be able to have CUE as your source of truth. Either actually (as in you maintain CUE files) or logically (you might import CUE from protobuf). But let's focus for now on the "simpler" case of maintaining CUE files as the source of truth for your types and certain constraints. myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716761360759?thread_ts=1694699710.517009&cid=CLT3ULF6C) The place we are looking to get to is roughly as follows: you will be able to take those CUE files and run something cue export *.cue --out go jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716796920309?thread_ts=1694699710.517009&cid=CLT3ULF6C) Yeah, that's exactly what I'm looking for. Something I can go:generate! myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716841456699?thread_ts=1694699710.517009&cid=CLT3ULF6C) That will at a minimum generate Go files that contain type declarations consistent with the CUE definitions, but it might also generate functions/methods that allow you to validate a "thing" in a way consistent with constraints that you have declared in CUE. The latter part is clearly optional if all you want is the types myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716937562299?thread_ts=1694699710.517009&cid=CLT3ULF6C) (even if we didn't generate a native Go encoding of the constraints, i.e. x: <5 ensuring that a valid thing has a field x with a value < 5, you could at this point, with just generated Go types, embed the CUE and actual run the CUE evaluator via the Go API on a value of the generated types. That would effectively be a 50/50 solution where the types are generated as native Go, but we rely on the CUE evaluator for the checking over the constraints against a Go value). myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716968567549?thread_ts=1694699710.517009&cid=CLT3ULF6C) Whilst we are not there yet today, that's the rough goal. myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694716976849109?thread_ts=1694699710.517009&cid=CLT3ULF6C) Yeah, that's exactly what I'm looking for. Something I can go:generate! Great myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694717046817219?thread_ts=1694699710.517009&cid=CLT3ULF6C) One option today is to use Go types as the source of truth, use cue get go to generate CUE definitions from the Go types, write the constraints in CUE that unify with the CUE definitions output from cue get go, and then use the Go API to evaluate the CUE constraints in much the same 50/50 approach I described above. myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694717072771049?thread_ts=1694699710.517009&cid=CLT3ULF6C) Not quite as clean, but you still effectively have CUE as a source of truth, albeit via the types initially being declared in Go. myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694717089756009?thread_ts=1694699710.517009&cid=CLT3ULF6C) I could sketch out a brief example if this would be helpful? myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694717133186749?thread_ts=1694699710.517009&cid=CLT3ULF6C) cc [@Jonathan Matthews](https://cuelang.slack.com/team/U03E1LP31AA) - this might be a good tutorial :+1: 1 :white_tick: 1 jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694717282222679?thread_ts=1694699710.517009&cid=CLT3ULF6C) That's an interesting approach. I think an example of that would be helpful. jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694717393063629?thread_ts=1694699710.517009&cid=CLT3ULF6C) So the source of truth would be Go source, but that could be output as a CUE file which could be used by the CLI? myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694718467462929?thread_ts=1694699710.517009&cid=CLT3ULF6C) Either use by the cmd/cue CLI, yes, and/or via the Go API. myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694718487914559?thread_ts=1694699710.517009&cid=CLT3ULF6C) Here is an example of using the result in the Go API, i.e. doing Go -> CUE -> Go "ish" myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694718494235899?thread_ts=1694699710.517009&cid=CLT3ULF6C) go mod tidy go generate ! go run . cmp stderr stderr.golden -- cue.mod/gen/mod.com/main_go_gen.cue -- // Code generated by cue get go. DO NOT EDIT. //cue:generate cue get go mod.com package person #Person: { Name: string Age: int } -- cue.mod/module.cue -- module: "mod.com" -- go.mod -- module mod.com go 1.20 require cuelang.org/go v0.6.0 -- main.go -- package main import ( "embed" "io/fs" "log" "path/filepath" "cuelang.org/go/cue" "cuelang.org/go/cue/cuecontext" "cuelang.org/go/cue/load" ) // The flow roughly looks like this: // // 1. Declare types in Go; we declare Person below // 2. Generate CUE definitions from Go types via // 'cue get go' (see go:generate directive below) // 3. Augment the generated CUE definitions with // whatever constraints you like; we do that in person.cue // 4. Embed the CUE files that make up the CUE person // package as an fs.FS // 5. Write a validator method/function which takes a value // of Go type Person, and uses the CUE definition loaded // via the embeded fs.FS to validate that value. //go:generate cue get go -p person . //go:embed *.cue var personCUE embed.FS type Person struct { Name string Age int } func main() { ctx := cuecontext.New() // cue/load does not yet understand io/fs.FS. It will do // but for now we have to "fake" things via an overlay // passed to cue/load.Instances() overlay := make(map[string]load.Source) fs.WalkDir(personCUE, ".", func(path string, d fs.DirEntry, err error) error { if err != nil { log.Fatal(err) } if !d.Type().IsRegular() { return nil } contents, err := personCUE.ReadFile(path) if err != nil { log.Fatal(err) } overlay[filepath.Join("/", path)] = load.FromBytes(contents) return nil }) // Now create a load config that uses that overlay conf := &load.Config{ Dir: "/", Overlay: overlay, } // Now load the CUE that is at the root of the overlay bps := load.Instances([]string{"."}, conf) // Extract the CUE value in the single CUE instance we loaded v := ctx.BuildInstance(bps[0]) // Grab the #Person definition personDef := v.LookupPath(cue.ParsePath("#Person")) // Create a value of the Go type Person paul := Person{ Name: "Paul", Age: 105, } // Encode that Go value as a CUE value to validate it paulCUE := ctx.Encode(paul) // Unify the now CUE value and the #Person definition; and // check for errors. if err := personDef.Unify(paulCUE).Err(); err != nil { log.Fatal(err) } } -- main_go_gen.cue.golden -- // Code generated by cue get go. DO NOT EDIT. //cue:generate cue get go mod.com package person #Person: { Name: string Age: int } -- person.cue -- package person // #Person is code generated via 'cue get go' and is // in scope by virtue of that definition being in // the same package, person. Hence, the constraints // we write below _augment_ that generated definition. #Person: { Age: <100 } -- stderr.golden -- main.go:86: #Person.Age: invalid value 105 (out of bound <100) exit status 1 myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694718511750439?thread_ts=1694699710.517009&cid=CLT3ULF6C) Hopefully the Go comments help to bring the example life. myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694718542971259?thread_ts=1694699710.517009&cid=CLT3ULF6C) I'm off to bed here, but in case the txtar file format above is new to you, take a look at https://github.com/cue-lang/cue/wiki/Creating-test-or-performance-reproducers myitcv [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694718563842769?thread_ts=1694699710.517009&cid=CLT3ULF6C) Note the CUE written/generated in this example could be used without modification via cmd/cue as well. jesselang [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694718745753019?thread_ts=1694699710.517009&cid=CLT3ULF6C) This is helpful. I'll explore this more. Thanks! Tony Worm [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694740500711979?thread_ts=1694699710.517009&cid=CLT3ULF6C) to put hof in context here the goal was to move the templates or translation out of the tool, to give users control. hof started because there was a project were we wanted to specify custom templates in a code gen process and the author didn't want to support that. hof gen is an equivalent of go:generate and can be invoked that way if desired. The main thing is that there is typically an extra cue file to define how the data should be mapped onto the code gen process, again more control when you need it. There are ways to run hof on a directory of templates, without that file, but you can also package that code gen (with a schema and templates) into a module, so you can rely on another module for handling the details of the code gen. we're very much in the camp that CUE should be the source of truth, mainly because the other codecs don't have the ability to specify concepts beyond their domain. CUE is very open ended and you can basically build any DSL you want on top of it. The code generators, and the schemas that define valid input to them, are effectively a DSL anyway. Tony Worm [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694740646429229?thread_ts=1694699710.517009&cid=CLT3ULF6C) So you would still have to maintain a CUE file for the data, and a CUE file for stub generation? given what I have just wrote You have your CUE that defines your types There is the code gen CUE file that defines how that type maps onto templates, in the same or more typically a separate module. Making the code generator separate means you can use it with multiple projects as a proper dependency, versioned and all those good things. For (2) someone has to maintain this, but the idea is the same with any language ecosystem, there are a few who maintain the projects that the many use to build other things Tony Worm [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694741039833769?thread_ts=1694699710.517009&cid=CLT3ULF6C) (2) by putting the code gen mapping in a CUE file (and using hof), you can do things like format the code during code gen, makes writing templates so much easier, no extra steps or tools to install (container based) run pre/post exec hooks, mainly a utility or convenience condition file creation based on your data model, often the data will contain more than just a data model that maps onto a database or types, you might want to capture things like the auth setup, or change how things are done because you use technology X or Y. You can generate a lot more from a data model plus a bit extra DSL write custom code directly in the output files, hof can use a diff and shadow to resolve changes Tony Worm [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694741300395009?thread_ts=1694699710.517009&cid=CLT3ULF6C) [@myitcv](https://cuelang.slack.com/team/UNVGZ9A2K) I really like the self-referentialism (strange loop?) in your example! very hofstadter of you Tony Worm [27 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694741607894959?thread_ts=1694699710.517009&cid=CLT3ULF6C) makes me think of the Contracrostipunctus (http://www.russellsteinberg.com/contracrostipunctus) myitcv [26 days ago](https://cuelang.slack.com/archives/CLT3ULF6C/p1694748770669579?thread_ts=1694699710.517009&cid=CLT3ULF6C) to put hof in context here This is great stuff, [@Tony Worm](https://cuelang.slack.com/team/UV68AGWNL) , thanks for completing the picture in this space with respect to hof. Just to reiterate, my comments were very deliberately limited to the current state of and plans for CUE itself, its encoders/decoders etc. hof has long been and remains a great project built on CUE, and more than that [@Tony Worm](https://cuelang.slack.com/team/UV68AGWNL) has been and remains an incredible member of the CUE community. Just that I'm very definitely not qualified to talk about the details of hof :slightly_smiling_face: ```
jpluscplusm commented 6 months ago

@myitcv This issue's title seems to me to be satisfied by https://cuelang.org/docs/concept/how-cue-works-with-go/#loading-non-cue-data. Do you feel this is sufficient, or is there more requested/implied by the thread that needs this issue to remain open?