Make layer less verbose

[pascalwhoop]

Hi y'all

we recently switched from

layer: foo

to

  metadata:
    kedro-viz:
      layer: 04_features

that's super verbose. Can we please get

  metadata.kedro-viz.layer: 01_raw

? Else the catalog just becomes suuuuper verbose. There must be a way to be yaml compliant when writing catalog yaml no?

[pascalwhoop]

I know this is probably kedro datasets originating, then let's fix it there :)

[astrojuanlu]

There must be a way to be yaml compliant when writing catalog yaml no?

The closest YAML-compliant, 1 liner I can think of that achieves the same structure is

ds:
  type: ...
  metadata: {"kedro-viz": {"layer": "04_features"}}

But that's even more annoying to type... (unsure if these are "Flow style" or "Block style" as per the YAML spec)

This is hardly a kedro-viz issue though, moving it to kedro-plugins for now

[noklam]

There must be a way to be yaml compliant when writing catalog yaml no?

I am happy if this exists, as of the moment the issue hasn't been addressed, see https://github.com/omry/omegaconf/issues/1188

Alternatlively, you can remove some typing with variable interpolation?

catalog.yml

_viz_feature_layer:
  kedro-viz:
    layer: 04_features

my_dataset:
  ...
  metadata: ${_viz_feature_layer}

?

[astrojuanlu]

Also see https://github.com/kedro-org/kedro-viz/issues/2127

[merelcht]

Discussed this in backlog grooming and we decided to not change anything about the layer definition. The metadata key was introduced to allow more flexible entries in the datacatalog for plugins (like Kedro-Viz).

[astrojuanlu]

And to clarify, there are ways to make this simpler, like YAML anchors https://github.com/kedro-org/kedro-viz/issues/2127 or even Omegaconf variables