Add a "map" statement

[kwatsen]

From https://tools.ietf.org/html/draft-openconfig-netmod-opstate-01#section-9.2:

9.2. YANG lists as maps

YANG has two list constructs, the 'leaf-list' which is similar to a list of scalars (arrays) in other programming languages, and the 'list' which allows a keyed list of complex structures, where the key is also part of the data values. As described in Section 8.1.2, the current requirements on YANG list keys require either duplication of data, or treating some data (i.e., those that comprise list keys) as a special case. One solution is to generalize lists to be more like map data structures found in most modern programming languages, where each list member has a key that is not required to part of the configuration or state data, and also not subject to existing "config-under-state limitations. This allows list keys to be arbitrarily defined by the user if desired, or based on values of data nodes. In the latter case, the specification of which data nodes are used in constructing the list key could be indicated in the meta-data associated with the key.

[rgwilton]

I had a slightly different way of thinking about this map type.

Containers can either have presence or just be structural organization of their child nodes.

But today YANG list entries always have presence, there is no way just to express the dual list equivalent to a "non presence" container.

Hence I was wondering whether this problem could be solved with a new "structural" YANG keyword:

• the structural keyword can be used on a list to indicate that it is structural and list entries do not imply presence just through existence.
• the structural keyword can also be used on a leaf to indicate that it is only structural and doesn't imply existence in configuration or state.

E.g. something along the lines of:

container interfaces {
  list interface {
    structural "Holds interface configuration & state";
    key "name";
    leaf name {
        structural "name of interface list entry";
    }

    container config {
      leaf configured;
      ...
    }

    container state {
      config false;
      leaf applied;
      ...
    }
  }
} 

[robshakir]

I would much prefer a "proper" map type, which maps to programming language concepts much more readily than some new YANG-specific means of specifying something is structural or not.

For the record, my email to a group of netmod@ subscribers suggested syntax akin to:

map interface {
  key {
    leaf name {
      type string;
      description
        "The name of the interface, as specified by the local system
        or by the operator";
    }
  }

  container config {
    ...
  }

  container state {
    config false;
    ...
  }
}

Composite keys would be readily supported by having multiple leaves specified within the key substmt. Programming languages creating bindings for such an object would map this to a data structure able to contain multiple values which keyed the map.

With an example YANG map of:

map tunnel {
  key {
    leaf source { 
      type inet:ip-address-no-zone;
    }
    leaf destination {
      type inet:ip-address-no-zone;
    }
  }
  ...
 }

In Python one might use either a dictionary or tuple to represent the key statement in this case. Another alternative might be to utilise a specific key object. This kind of approach is likely possible in numerous other languages (should be very do-able in Java):

tunnel[{'source': "192.0.2.1", 'destination': "192.0.2.2}] = ...
tunnel[("192.0.2.1", "192.0.2.2")] = ...

class TunnelKey:
   def __init__(self, src, dst):
     self.source = src
     self.destination = dst

tunnel[TunnelKey(src,dst)] = ...

Languages that do not have such objects could use a struct:

type TunnelContents struct {
  ...
}

type TunnelKey struct {
   source        string
   destination   string 
}

var tunnel map[TunnelKey]TunnelContents
tunnel[TunnelKey{source: "2001:db8::1", destination: "2001:db8:CAFE::2"}] = TunnelContents{}

In both these cases, there is a significant benefit in terms of such mappings because we do not need to create the whole TunnelContents struct/contained object in the Python/Java case to then test the key values that are supplied, the data structure that is used for the key of the list is much lighter weight and can be easily initialised separately. Embedding the key inside the value (which I think would still be implied with structural) does not have this advantage.

The other benefit of using a map type is it keeps things distinct from the existing list, which I believe that there will still be a requirement for, and we can have distinct mappings for serialisation formats. Map should end up being represented by a JSON object, whereas list should be a JSON array. The structural overloading above adds complexity to this distinction.

r.

[mbj4668]

@robshakir - In your proposal there is a new syntax for keys, and some new semantics. IMO your proposed syntax could work with the current YANG 1/1.1 semantics, OR the current YANG 1/1.1 syntax could work with your new proposed semantics.

Also, it is not clear to me what exactly the new semantics is. You can certainly use a Python dict with a tuple for the combined keys today with the current YANG semantics for keys.

[robshakir]

@mbj4668 - I don't think that this is true. In Y1/1.1, then the key must be somewhere within the list entry (at the top level). This proposal changes this such that we have something more akin to a common map, where there is no coupling between the key of the map and the value that it contains.

The intention of showing the programming language mappings is that I feel that the current list type is very unnatural when it comes to being mapped to common languages.

[kwatsen]

Rob's example above can be distilled down to:

  map foo {
    key {
      // just leaf statements?
      }
    // arbitrary data nodes here (container, leaf, leaf-list, etc.)
  }

From instance encoding perspective, would we presumably have:

  <interface>
    <key>
      <name>bar</name>
    </key>
    <config>...</config>
    <state>...</state>
  </interface>

Or would it be better to have an explicit 'value' container to eliminate possible name collisions:

  <interface>
    <key>
      <name>bar</name>
    </key>
    <value>
      <config>...</config>
      <state>...</state>
    </value>
  </interface>

And if that's the case, maybe the YANG should be:

  map foo {
    key {
      // just leaf statements?
      }
    value {
      // arbitrary data nodes here (container, leaf, leaf-list, etc.)
    }
  }

My assumption is that the 'key' would be configuration just as much as the 'value'. That is, CRUD operations could act on the key alone. so long as some uniqueness validation constraint is ensured.

[rgwilton]

I'm trying to get my head around this.

I think that I agree with Martin that a YANG list can be easily be implemented as a map from the set of keys to an object that probably also holds those same keys along with any other list entry data. In fact, I naively assume that generally YANG lists would be implemented as [sorted] maps to allow for efficient look ups.

So, I'm trying to think of what the problems with the existing YANG lists are. I think that they may be:

• Lists cannot hold a combination of config and state entries since the keys must either belong to config or state (which is the problem that I was trying to solve with my structural keyword).
  • List keys must be present in the top level of the list entries.

Is the proposed map construct expected to solve both of these problems?

[robshakir]

The fundamental difference is to decouple the key of the map from the value entirely.

Implementing YANG lists as sorted maps has some overhead that feels clumsy to me.

1. One needs to track changes to the map keys and reflect those in the value.
2. One needs to instantiate the contained structure/object (the value of the map) to check that the key values are valid.

r.

[robshakir]

Both the issues that you list would be solved based on this map because:

1. We have removed the need to have the key anywhere in the list.
2. The value simply does not appear in the contained list entry.

It would be possible to have it appear using some form of alias/pointer type - I'd propose that this be solved using something like:

map a-map {
  key {
   leaf key-one { ... }
   leaf key-two { ... }
   in-value <relative-path>;
 }
}

If there is sufficient demand for it. The "reflected" leaves reflecting the key inside the value would not be directly writable.

r.

[schoenw]

Sounds like a non-trivial proposal since many details need to be worked out. I suggest to defer.

[mbj4668]

@mbj4668 - I don't think that this is true. In Y1/1.1, then the key must be somewhere within the list entry (at the top level).

In the YANG definition yes, but not internally in the implementation. For example, we don't store the key values twice in our implementation; they are just present as keys, not in the "body" of the entry.

I think that this is an implementation detail - with the current syntax I can store a list in a map, and with the new proposed syntax I can store a "YANG map" in a list.

But I still don't understand if the proposal changes some semantics compared to the "list" statement?

This proposal changes this such that we have something more akin to a common map, where there is no coupling between the key of the map and the value that it contains.

Yes, I understand that the proposed syntax makes it clear that the keys are separate from the rest of the values.

[kwatsen]

I recently had to model a REST API using JSON Schema.

Two things impressed me about this effort:

1. YANG (w/ RESTCONF) is much better at defining a REST API
2. However, JSON Schema did a very good job supporting maps

Note that the JSON structure was inherited. Another way of saying this, this is the JSON that the developers wanted without having to fit into YANG.

To give an idea, here are two structures. In both cases note that the "key" is a variable. Using the syntax Rob provides above, this is to say that leaf name type string has a pattern for allowed values.

Examples 1

{
    "link_data": {
        "left1, right1": {
            "loss_db": {
                "value": 0.33,
                "timestamp": "2023-11-30T18:12:00Z",
                "how_obtained": "MEASURED",
            }
        },
        "left2, right2": {
            "distance_m": {
                "value": 0.33,
                "timestamp": "2023-11-30T18:12:00Z",
                "how_obtained": "MEASURED",
            }
        },
    }
}

Example 2:

    {    
         "nodes": {
             "31b6f6c0-8c8b-42ce-990e-3092d6163d4a": {
                 "coordinates": {"x": 255, "y": 150}
             },
             "71cb1dae-bca0-4551-bcae-12103e9f2f97": {
                 "coordinates": {"x": 105, "y": 120}
             },
             "b82e8192-8276-4fe8-8d8a-a610b933a871": {
                 "coordinates": {"x": 420, "y": 225}
             },
         }
     } 

The above examples are in JSON, which maps directly to Python. How the same would map to XML is unclear but, since no-one maps XML to Python, it can probably be whatever is minimally sufficient.

[abierman]

I do not really understand the proposed changes. Agree with Not Now

[kwatsen]

Proposal needs to explain how lists and maps work together

[plajjan]

I don't think this can ever be added to YANG. I get how YANG lists might not map perfectly to maps in programming languages, but it's really too late to do anything about that now, even in YANG 2.0. Adding a map to YANG doesn't improve things since all the existing YANG models using list still need to be handled. It wouldn't remove ambiguity, just add more complexity.

Anyone (I count myself to this group) implementing bindings / mapping to a programming language need to handle lists, both keyed and non-keyed (for oper state). If you've already done this, we're already done, no? We can't pull another NMDA and change all the models in the world IMHO

[robshakir]

Certainly the ship has likely sailed. Unsurprising for a 10 year old discussion.

Both Thrift and protobuf navigated this discussion and the coexistence of a map and list type a number of years ago (prototype definitely added it later, I can't quite recall whether the same happened in Thrift).

I will refrain from commenting about the velocity of addressing feature requests here, and agree that the last thing that YANG needs is more complexity.