cue/load: Build Attributes and Directory Hierarchies

What version of CUE are you using (`cue version`)?

$ cue version
cue version v0.4.3 darwin/arm64

Does this issue reproduce with the latest release?

Yes.

What did you do?

Directory hierarchies are currently being used to be able to build up configurations that have environmental, regional, and other differences. A typical directory structure would look something similar to:

.
├── development
│   └── echo.cue
├── echo.cue
└── production
    ├── echo.cue
    ├── osaka
    │   └── echo.cue
    └── tokyo
        └── echo.cue

Build attributes would eliminate the need for specific structures; but there are others which might be necessary, and users might want to keep the typical structure as a means of organization. I don't want to be opinionated about how users organization their configuration.

When experimenting with build attributes and directory hierarchies, behavior when exporting was unexpected.

The following is a simple reproducer, which matches the above directory structure:

-- cue.mod/module.cue --
module: "acme.com"
-- development/echo.cue --
@if(dev)

package echo

Metadata: {
    environment: "development"
}

Application: {
    spec: {
        resources: {
            requests: {cpu: 0.5, memory: 128M}
            limits: {cpu: 0.5, memory: 128M}
        }

        replicas: 1
    }
}
-- echo.cue --
package echo

Metadata: {
    serviceID: "echo-jp"
}

Application: {
    spec: image: name: "echo"
}
-- production/echo.cue --
@if(prod)

package echo

Metadata: {
    environment: "production"
}

Application: {
    spec: resources: {
        requests: {cpu: 1, memory: 256M}
        limits: {cpu: 1, memory: 256M}
    }
}
-- production/osaka/echo.cue --
@if(prod && osaka)

package echo

Metadata: {
    region: "osaka"
}

Application: {
    spec: replicas: 3
}
-- production/tokyo/echo.cue --
@if(prod && tokyo)

package echo

Metadata: {
    region: "tokyo"
}

Application: {
    spec: replicas: 5
}

What did you expect to see?

A single set of merged structures:

$ cue export -t prod -t tokyo ./...
{
    "Metadata": {
        "serviceID": "echo-jp",
        "environment": "production",
        "region": "tokyo"
    },
    "Application": {
        "spec": {
            "image": {
                "name": "echo"
            },
            "resources": {
                "requests": {
                    "cpu": 1,
                    "memory": 256000000
                },
                "limits": {
                    "cpu": 1,
                    "memory": 256000000
                }
            },
            "replicas": 5
        }
    }
}

What did you see instead?

Multiple sets of strcutures, one for each level of the directory hierarchy:

$ cue export -t prod -t tokyo ./...
{
    "Metadata": {
        "serviceID": "echo-jp"
    },
    "Application": {
        "spec": {
            "image": {
                "name": "echo"
            }
        }
    }
}
{
    "Metadata": {
        "serviceID": "echo-jp",
        "environment": "production"
    },
    "Application": {
        "spec": {
            "image": {
                "name": "echo"
            },
            "resources": {
                "requests": {
                    "cpu": 1,
                    "memory": 256000000
                },
                "limits": {
                    "cpu": 1,
                    "memory": 256000000
                }
            }
        }
    }
}
{
    "Metadata": {
        "serviceID": "echo-jp",
        "environment": "production",
        "region": "tokyo"
    },
    "Application": {
        "spec": {
            "image": {
                "name": "echo"
            },
            "resources": {
                "requests": {
                    "cpu": 1,
                    "memory": 256000000
                },
                "limits": {
                    "cpu": 1,
                    "memory": 256000000
                }
            },
            "replicas": 5
        }
    }
}

@slewiskelly thanks for raising this. The issue here is that ./... matches the current directory and its subdirectories. We don't have a pattern for specifying "all the leaf instances of echo", which is I think what you're intending here (please let me know if that's not the case though). i.e. you want to select all directories below the current directory which contain a package echo, where that directory itself does not contain a subdirectory that is also a package echo (ignoring build constraints). According to the structure you have laid out above, that currently corresponds to the glob pattern ./*/*/*, but might not be limited to that, i.e. there might be times when a leaf instances sits at level 1 (where 0 is the current directory).

There are at least two ways I can think of achieving this:

Providing a more specific pattern to enable you to "select" just the leave nodes.
Changing the behaviour of export (or adding a flag) so that it drops incomplete packages in this load mode, on the understanding that non-leaf directories would be incomplete.

Option 2 feels like it will create a footgun, because packages that unintentionally contain incomplete errors will be dropped when they shouldn't be.

Option 1 feels doable, either via something like a glob (which would clash with the shell unless quoted) or via a more nuanced selector/flag like (just chucking ideas out there):

cue export -L ./...:echo

Here the -L flag would select only leaf packages, and :echo would limit to the package echo.

On a related point, how are you finding this loading pattern of having a single package span multiple directories? We definitely have people in two quite distinct camps when it comes to this pattern!

cue-lang / cue