hofstadter-io / hof

Framework that joins data models, schemas, code generation, and a task engine. Language and technology agnostic.
https://docs.hofstadter.io
Apache License 2.0
519 stars 36 forks source link

hof/gen: file inputs values are merged in an anti-CUE way #205

Open verdverm opened 1 year ago

verdverm commented 1 year ago

This is left over behavior from the days before CUE, but we can do better now.

package gen

// A file which should be generated by hof
#File: {

    // The local input data, any struct
    // The Generator.In will be merged here
    //   but will not replace any values set locally
    In?: {...} // for templates

    // input value for data files, always remains a CUE value
    Val?: {...} | *In // for data files

Within the generator code, while decoding a generator, when filling the In for the file, we merge the generator In, by looping over fields and adding them if they are not found. This is not the CUE way. It was done as a convenience, or maybe performance?

We should handle this in the generator schema

// Definition for a generator
#Generator: {

    // Generator wide input value, typically authors provide additional named inputs
        // and use this field to provide a shared, reshaped, validated input to all output files.
    In: {...}

    // Should In be added to the input of every output file?
    applyInToAllOut: bool | *true
    if applyInToAllOut == true {
        Out: [...{"In": In}]
    }

If this conditional pattern constraint causes a performance issue, we can handle it in code by manually doing the join on the cue value before decoding into tho Go map.

verdverm commented 1 year ago

So yea, doing this in the schema crushes performance bad. Let's not.

We should try merging just before file In decode, if needed, and see if the performance takes a hit. If not, we can use the flag, and the File local flag for override. The local flag will allow users to apply settings to different sublists of output files, and then combine them for the final Out file list.

The merge actually happens with Go maps and concrete values, so maybe we need a recursive merge that reports an error when values collide, but will be ok if the same. Depends on the results of a local CUE merge performance impact.