Original thread (click to expand)
#### Andy Walker
Have an interesting issue with pattern constraints. Not sure if I should submit a bug or if I’m doing something incorrectly. I am attempting to keep the value definitions of a config file format in Cue separate from the validation, and have split it up into two files, which I’m compiling and validating using the Go API before encoding into a struct. I am defining a map constraint like so, where the idea is that there must be three (and only three) top level keys, each of which are themselves maps of strings. I am attempting to constrain the fields such that they must match the regexp `^[a-zA-Z0-9_-]+$`:
```cue
vars: close({
"a": #varsSchema
"b": #varsSchema
"c": #varsSchema
})
#varsSchema: {
[=~"^[a-zA-Z0-9_-]+$"]: string & !=""
}
vars: {
a: {
"foo.1": "1"
}
}
// error: vars.a."foo.1": field not allowed
````
As expected, this example throws an error since `foo.1` is not a valid field name. However, if I split this into a “schema” and a value and compile them separately using `cue.Scope` for the value, it does not check this constraint. Full code in thread.
```go
package main
import (
"log"
"cuelang.org/go/cue"
"cuelang.org/go/cue/cuecontext"
)
const cueCombined = `
vars: close({
"a": #varsSchema
"b": #varsSchema
"c": #varsSchema
})
#varsSchema: {
[=~"^[a-zA-Z0-9_-]+$"]: string & !=""
}
vars: {
a: {
"foo.1": "1"
}
}`
const cueSchema = `
vars: close({
"a": #varsSchema
"b": #varsSchema
"c": #varsSchema
})
#varsSchema: {
[=~"^[a-zA-Z0-9_-]+$"]: string & !=""
}
`
const cueValue = `
vars: {
a: {
"foo.1": "1"
}
}`
func main() {
log.SetFlags(0)
cctx := cuecontext.New()
combined := cctx.CompileString(cueCombined)
if combined.Err() != nil {
log.Printf("combined compile: %v", combined.Err())
}
cctx2 := cuecontext.New()
schema := cctx2.CompileString(cueSchema)
if schema.Err() != nil {
log.Fatalf("schema compile: %v", schema.Err())
}
value := cctx2.CompileString(cueValue, cue.Scope(schema))
if value.Err() != nil {
log.Fatalf("value compile: %v", value.Err())
}
if err := value.Validate(); err != nil {
log.Fatalf("value.Validate: %v", err)
}
log.Println("scope version all good?")
}
```
Output:
```
combined compile: vars.a."foo.1": field not allowed
scope version all good?
```
The codec does not complain either, FWIW, and the field is encoded into the resultant `map[string]string` as if the constraint was not specified.
#### eonpatapon
I think you need to do `value.Unify(schema)` before `value.Validate()`
#### Andy Walker
Alas, that did not solve it
#### myitcv
The issue was actually your regexp - it did not allow for the literal `.`
```
go mod tidy
! go run .
cmp stderr stderr.golden
-- go.mod --
module mod.com
go 1.21.4
require cuelang.org/go v0.7.0
-- main.go --
package main
import (
"fmt"
"log"
"cuelang.org/go/cue/cuecontext"
)
const cueSchema = `
vars: close({
"a": #varsSchema
"b": #varsSchema
"c": #varsSchema
})
#varsSchema: {
[=~#"^[a-zA-Z0-9_\-]+$"#]: string & !=""
}
`
const cueValue = `
vars: {
a: {
"foo.1": "1"
}
}`
func main() {
// Load all values in the same context
ctx := cuecontext.New()
// Load the schema from somewhere
schema := ctx.CompileString(cueSchema)
if err := schema.Err(); err != nil {
log.Fatalf("schema compile: %v", err)
}
// Load the value from somewhere else
value := ctx.CompileString(cueValue)
if err := value.Err(); err != nil {
log.Fatalf("value compile: %v", err)
}
// Validate the value by first unifying with the schema and then checking
// the error of the resulting value
value = value.Unify(schema)
if err := value.Validate(); err != nil {
log.Fatalf("value.Validate: %v", err)
}
fmt.Printf("%v\n", value)
}
-- stderr.golden --
main.go:49: value.Validate: vars.a."foo.1": field not allowed
exit status 1
```
Hopefully that solves it for you, Andy Walker?
#### Andy Walker
No, alas not, the issue is actually the opposite. I want that error, but I am not getting it when I feel like I should. The example in the thread might not be the greatest, but where it says “scope version all good?” I am actually expecting an error (edited)
unless I am misreading your solution! always possible :slightly_smiling_face:
For context, this is me attempting to use raw CUE as a config file with an internally validated schema
which… may not be the intended use?
#### myitcv
Ah
My bad
I've just updated my example
That should be what you are after?
#### Andy Walker
Will test! If that kicks back the error in both circumstances, it is indeed what I’m after! Mostly I wanted to be able to configure a service using all of the great, consistent dsl-y features of Cue without needing a confusing import or schema code inside the config.
#### myitcv
Yep, that's fully possible
Indeed what we do in some situations is create some self-contained CUE that is derived from some more human-readable CUE that does have imports etc
i.e. it's often awkward to not use imports where they are more natural
#### Andy Walker
That said, the big question in technology is always “am I trying to hammer screws?“, so if this method is fighting the tech or there’s a better, more idiomatic way this should be done, I’m interested to know
#### myitcv
To build more on the situation I am describing, we embed the resulting self-contained CUE in a Go binary
If that kicks back the error in both circumstances
You already had it working on one of the cases I think, so I just showed how to do the other case :slightly_smiling_face:
#### Andy Walker
Oh! while I have you
(sorry, looking up the CUE). But I am attempting to make a “var” that can be referenced like a macro where, at runtime, the “common” part of the vars map is merged with the os-specific one into a var value but this requires the injection of a single value at runtime, and I’m unsure how to do it. One moment, the example will be more clear
```cue
vars: close({
"common": #varsSchema
"windows": #varsSchema
"mac": #varsSchema
"linux": #varsSchema
})
#varsSchema: {
[=~"^[a-zA-Z0-9_-]+$"]: string & !=""
}
#currentPlatform: string //set during evaluation
var: vars.common&vars[#currentPlatform]
```
So in the code above, I would make `#currentPlatform` concrete somewhere between loading the schema and the config
Wasn’t sure how to do that
(I would be synthesizing it from runtime.GOOS)
#### myitcv
You're likely looking for injection of values: https://alpha.cuelang.org/docs/reference/cli/cue-injection/
Specifically tag variables
#### Andy Walker
Correct. currently I’m simply concatting the bytes:
```go
var osValue string
switch runtime.GOOS {
case "windows":
osValue = `#currentPlatform:"windows"`
case "linux":
osValue = `#currentPlatform:"linux"`
case "darwin":
osValue = `#currentPlatform:"mac"`
}
config := ctx.CompileString(
strings.Join([]string{string(f), osValue, schema}, "\n"),
cue.Filename(path),
)
```
I wasn’t able to find where to do injection with the API
#### myitcv
See `TagVars` on https://pkg.go.dev/cuelang.org/go/cue/load#Config
(or you could just use cue.Value.FillPath())
#### Andy Walker
would that be
`schema.Value().FillPath(cue.MakePath(), val)`
or would I do it on the value before merging? Or does it matter?
```go
func Load(path string) (*Plan, error) {
f, err := os.ReadFile(path)
if err != nil {
return nil, err
}
osValue := runtime.GOOS
if osValue == “darwin” {
osValue = “mac”
}
ctx := cuecontext.New()
// Load the schema from somewhere
schema := ctx.CompileString(schema)
if err := schema.Err(); err != nil {
return nil, fmt.Errorf(“schema compile: %v”, err)
}
schema.Value().FillPath(cue.MakePath(cue.Str(“#currentPlatform”)), osValue)
// Load the value from somewhere else
config := ctx.CompileBytes(f, cue.Scope(schema), cue.Filename(path))
if err := config.Err(); err != nil {
return nil, fmt.Errorf(“value compile: %v”, err)
}
// Validate the value by first unifying with the schema and then checking
// the error of the resulting value
config = config.Unify(schema)
if err := config.Validate(); err != nil {
return nil, fmt.Errorf(“value.Validate: %v”, err)
}
codec := gocodec.New(ctx, nil)
cfg := &Plan{}
if err := codec.Encode(plan, cfg); err != nil {
println(“encode”)
return nil, cErr(err)
}
return cfg, nil
}
```
but it’s tossing back
`main.go:18: var: invalid non-ground value string (must be concrete string)`
I expect I’m using `cue.MakePath` and `cue.Str` Incorrectly. I’ll put back my dedicated error wrapping code and see if it gives me a more precise error.
Or, I guess `cue.Def`, rather
So the current attempt is to use fillpath on the compiled schema BEFORE processing the value(config).
```go
schema := ctx.CompileString(schema)
if err := schema.Err(); err != nil {
return nil, fmt.Errorf("schema compile: %v", err)
}
schema = schema.Value().FillPath(cue.MakePath(cue.Def("currentPlatform")), osValue)
config := ctx.CompileBytes(f, cue.Scope(schema), cue.Filename(path))
if err := config.Err(); err != nil {
return nil, fmt.Errorf("config compile: %v", err)
}
```
Unlike attempting `cue.MakePath` on the compiled config, this solves the concrete value problem, so `#currentPlatform` is getting set; however, now the fields are gone. For reference:
```cue
// relevant schema
vars: close({
"common": #varsSchema
"windows": #varsSchema
"mac": #varsSchema
"linux": #varsSchema
})
#currentPlatform: string //set during evaluation
var: vars.common&vars[#currentPlatform]
// config
vars: {
"common": {
"testHash": "foo"
}
windows: {
"target_dir": #"C:\dir1"#
}
mac: {
target_dir: "/dir2"
}
linux: {
target_dir: "/dir3"
}
}
plan: {
actions: [
createFile&{path: "foo", from: var.testHash},
]
}
```
Results in: `plan.actions.0.from: undefined field: testHash`
So, it seems something’s getting overwritten, and I am wondering if that is because the unification for var is not re-evaluated, however I’ve attempted to re-evaluate it to no avail.
It seems like TagVars might be the right approach (i.e. injecting values ahead of time), but I can’t seem to find a way to get from what I’ve got to cue/load
All of this works if I just concatenate schema bytes -> `#currentPlatform: %s\n` -> config bytes into a `string` or `[]byte` of course (edited)
I have been digging around in the source for awhile now and the best I can come up with so far is:
```go
sl := load.FromString(schema)
i := load.Instances([]string{"schema.cue"}, &load.Config{
// Tags: []string{"#currentPlatform=" + osValue},
TagVars: map[string]load.TagVar{
"#currentPlatform": {
Func: func() (ast.Expr, error) {
return ast.NewString(osValue), nil
},
},
},
Overlay: map[string]load.Source{
"schema.cue": sl,
},
})[0]
if i.Err != nil {
return nil, fmt.Errorf("load instance: %v", cErr(i.Err))
}
```
Which is pretty brutal and still doesn’t work because Overlay does not do what I thought it did.
Is there any other way?
I’ve combed all through Github, and pretty much every time I run across `load.Config.TagVars` or `load.Config.Tags`, it’s so deeply abstracted I’m not sure how to get it back to the context again, or if it references an entirely separate file-hierarchy-only based workflow that I’m just not intended to use for this. Similarly `Value.FillPath()` usage has proved surprisingly difficult to locate a good example of. (edited)
#### myitcv
Andy Walker - my apologies for the delay in replying here.
Is there some way you can share with me the code you're referring to above?
Probably easiest at this point to debug files on disk
In the meantime, let me throw together a quick example of each
Here's a `FillPath()` example
```
go mod tidy
go run .
cmp stdout stdout.golden
-- go.mod --
module mod.example
go 1.22.1
require cuelang.org/go v0.7.1
-- main.go --
package main
import (
"fmt"
"cuelang.org/go/cue"
"cuelang.org/go/cue/cuecontext"
"cuelang.org/go/cue/load"
)
func main() {
ctx := cuecontext.New()
// Load CUE from the current directory
bps := load.Instances([]string{"."}, nil)
before := ctx.BuildInstance(bps[0])
after := before.FillPath(cue.ParsePath("x"), 5)
fmt.Printf("before: %v\n", before)
fmt.Printf("after: %v\n", after)
}
-- x.cue --
package example
x: int
-- stdout.golden --
before: {
x: int
}
after: {
x: 5
}
```
Here is an example using Tags:
```
go mod tidy
go run .
cmp stdout stdout.golden
-- go.mod --
module mod.example
go 1.22.1
require cuelang.org/go v0.7.1
-- main.go --
package main
import (
"fmt"
"cuelang.org/go/cue/cuecontext"
"cuelang.org/go/cue/load"
)
func main() {
ctx := cuecontext.New()
// Load CUE from the current directory
bps := load.Instances([]string{"."}, &load.Config{
Tags: []string{"x=5"},
})
v := ctx.BuildInstance(bps[0])
fmt.Printf("v: %v\n", v)
}
-- x.cue --
package example
x: int @tag(x, type=int)
-- stdout.golden --
v: {
x: 5
}
```
TODO: figure out what docs could be derived from this long thread's multiple topics.
Slack thread: https://cuelang.slack.com/archives/CLT3ULF6C/p1706114386578579
Original thread (click to expand)
#### Andy Walker Have an interesting issue with pattern constraints. Not sure if I should submit a bug or if I’m doing something incorrectly. I am attempting to keep the value definitions of a config file format in Cue separate from the validation, and have split it up into two files, which I’m compiling and validating using the Go API before encoding into a struct. I am defining a map constraint like so, where the idea is that there must be three (and only three) top level keys, each of which are themselves maps of strings. I am attempting to constrain the fields such that they must match the regexp `^[a-zA-Z0-9_-]+$`: ```cue vars: close({ "a": #varsSchema "b": #varsSchema "c": #varsSchema }) #varsSchema: { [=~"^[a-zA-Z0-9_-]+$"]: string & !="" } vars: { a: { "foo.1": "1" } } // error: vars.a."foo.1": field not allowed ```` As expected, this example throws an error since `foo.1` is not a valid field name. However, if I split this into a “schema” and a value and compile them separately using `cue.Scope` for the value, it does not check this constraint. Full code in thread. ```go package main import ( "log" "cuelang.org/go/cue" "cuelang.org/go/cue/cuecontext" ) const cueCombined = ` vars: close({ "a": #varsSchema "b": #varsSchema "c": #varsSchema }) #varsSchema: { [=~"^[a-zA-Z0-9_-]+$"]: string & !="" } vars: { a: { "foo.1": "1" } }` const cueSchema = ` vars: close({ "a": #varsSchema "b": #varsSchema "c": #varsSchema }) #varsSchema: { [=~"^[a-zA-Z0-9_-]+$"]: string & !="" } ` const cueValue = ` vars: { a: { "foo.1": "1" } }` func main() { log.SetFlags(0) cctx := cuecontext.New() combined := cctx.CompileString(cueCombined) if combined.Err() != nil { log.Printf("combined compile: %v", combined.Err()) } cctx2 := cuecontext.New() schema := cctx2.CompileString(cueSchema) if schema.Err() != nil { log.Fatalf("schema compile: %v", schema.Err()) } value := cctx2.CompileString(cueValue, cue.Scope(schema)) if value.Err() != nil { log.Fatalf("value compile: %v", value.Err()) } if err := value.Validate(); err != nil { log.Fatalf("value.Validate: %v", err) } log.Println("scope version all good?") } ``` Output: ``` combined compile: vars.a."foo.1": field not allowed scope version all good? ``` The codec does not complain either, FWIW, and the field is encoded into the resultant `map[string]string` as if the constraint was not specified. #### eonpatapon I think you need to do `value.Unify(schema)` before `value.Validate()` #### Andy Walker Alas, that did not solve it #### myitcv The issue was actually your regexp - it did not allow for the literal `.` ``` go mod tidy ! go run . cmp stderr stderr.golden -- go.mod -- module mod.com go 1.21.4 require cuelang.org/go v0.7.0 -- main.go -- package main import ( "fmt" "log" "cuelang.org/go/cue/cuecontext" ) const cueSchema = ` vars: close({ "a": #varsSchema "b": #varsSchema "c": #varsSchema }) #varsSchema: { [=~#"^[a-zA-Z0-9_\-]+$"#]: string & !="" } ` const cueValue = ` vars: { a: { "foo.1": "1" } }` func main() { // Load all values in the same context ctx := cuecontext.New() // Load the schema from somewhere schema := ctx.CompileString(cueSchema) if err := schema.Err(); err != nil { log.Fatalf("schema compile: %v", err) } // Load the value from somewhere else value := ctx.CompileString(cueValue) if err := value.Err(); err != nil { log.Fatalf("value compile: %v", err) } // Validate the value by first unifying with the schema and then checking // the error of the resulting value value = value.Unify(schema) if err := value.Validate(); err != nil { log.Fatalf("value.Validate: %v", err) } fmt.Printf("%v\n", value) } -- stderr.golden -- main.go:49: value.Validate: vars.a."foo.1": field not allowed exit status 1 ``` Hopefully that solves it for you, Andy Walker? #### Andy Walker No, alas not, the issue is actually the opposite. I want that error, but I am not getting it when I feel like I should. The example in the thread might not be the greatest, but where it says “scope version all good?” I am actually expecting an error (edited) unless I am misreading your solution! always possible :slightly_smiling_face: For context, this is me attempting to use raw CUE as a config file with an internally validated schema which… may not be the intended use? #### myitcv Ah My bad I've just updated my example That should be what you are after? #### Andy Walker Will test! If that kicks back the error in both circumstances, it is indeed what I’m after! Mostly I wanted to be able to configure a service using all of the great, consistent dsl-y features of Cue without needing a confusing import or schema code inside the config. #### myitcv Yep, that's fully possible Indeed what we do in some situations is create some self-contained CUE that is derived from some more human-readable CUE that does have imports etc i.e. it's often awkward to not use imports where they are more natural #### Andy Walker That said, the big question in technology is always “am I trying to hammer screws?“, so if this method is fighting the tech or there’s a better, more idiomatic way this should be done, I’m interested to know #### myitcv To build more on the situation I am describing, we embed the resulting self-contained CUE in a Go binary If that kicks back the error in both circumstances You already had it working on one of the cases I think, so I just showed how to do the other case :slightly_smiling_face: #### Andy Walker Oh! while I have you (sorry, looking up the CUE). But I am attempting to make a “var” that can be referenced like a macro where, at runtime, the “common” part of the vars map is merged with the os-specific one into a var value but this requires the injection of a single value at runtime, and I’m unsure how to do it. One moment, the example will be more clear ```cue vars: close({ "common": #varsSchema "windows": #varsSchema "mac": #varsSchema "linux": #varsSchema }) #varsSchema: { [=~"^[a-zA-Z0-9_-]+$"]: string & !="" } #currentPlatform: string //set during evaluation var: vars.common&vars[#currentPlatform] ``` So in the code above, I would make `#currentPlatform` concrete somewhere between loading the schema and the config Wasn’t sure how to do that (I would be synthesizing it from runtime.GOOS) #### myitcv You're likely looking for injection of values: https://alpha.cuelang.org/docs/reference/cli/cue-injection/ Specifically tag variables #### Andy Walker Correct. currently I’m simply concatting the bytes: ```go var osValue string switch runtime.GOOS { case "windows": osValue = `#currentPlatform:"windows"` case "linux": osValue = `#currentPlatform:"linux"` case "darwin": osValue = `#currentPlatform:"mac"` } config := ctx.CompileString( strings.Join([]string{string(f), osValue, schema}, "\n"), cue.Filename(path), ) ``` I wasn’t able to find where to do injection with the API #### myitcv See `TagVars` on https://pkg.go.dev/cuelang.org/go/cue/load#Config (or you could just use cue.Value.FillPath()) #### Andy Walker would that be `schema.Value().FillPath(cue.MakePath(TODO: figure out what docs could be derived from this long thread's multiple topics.