Adding Support for Tag Directives

[jess-sol]

Hey all,

I'm working on a project that requires support for parsing Yaml documents with shorthand tag directives. I'm interested in looking into adding this support for yaml_rust, instead of writing my own Yaml parser. That said I don't have much experience with the yaml-rust project, or writing parsers in general.

So I wanted to start a discussion on what this feature should look like, how it should function, and how it may be implemented.

[chyh1990]

This library currently support several standard tag directives, e.g. !!str. But this feature is seldom used, can you give me some typical use cases? And how other YAML libraries support this feature (for example, in python or ruby) ?

[jess-sol]

Sure, I need to parse a Yaml CloudFormation template, which may or may not have local tags in it. I need to replace some of those tags (and their values) with new values; but some tags I just need to leave alone, so they'll be the same when I dump the Yaml. Here's a really simple example from one of my CloudFormation files:

WebInstance:
    Type: AWS::EC2::Instance
    CreationPolicy:
        ResourceSignal:
            Region: !Ref AWS::Region
            Timeout: !GetAtt ["a", "b", "c"]

I looked at how Ruby and Python handle custom tags. Tags are used to express the type of next node in the Yaml document, Ruby and Python both expose constructor functions that let the user register a function to be called when a specific tag is encountered, and the result of this function is then used as the value of the tagged node.

In Ruby:

require 'yaml'
YAML.add_domain_type '', 'Ref' do |type,val| { "Reference": val } end
thing = YAML.load_file('../../small-test.yaml')
puts thing.inspect

Which returns

{"WebInstance"=>{"Type"=>"AWS::EC2::Instance", "CreationPolicy"=>{"ResourceSignal"=>{"Region"=>{:Reference=>"AWS::Region"}, "Timeout"=>["a", "b", "c"]}}}}

There are actually four different methods in Ruby's Yaml module: add_builtin_type, add_domain_type, add_private_type, add_ruby_type Docs. They all do pretty much the same thing though, the major difference is the prefix of the tag (if it's a global tag from yaml.org, a custom domain, a private tag, or a global tag from ruby.yaml.org, respectively).

Ruby's implementation seems a bit wonky, they don't explicitly allow you to add local tags (aka !Ref), so instead you must use add_domain_type without a domain.

In Python:

import yaml
from pprint import pprint

class CFReference:
    def __init__(self, reference):
        self.reference = reference

class CFGetAtt:
    def __init__(self, attribute):
        self.attribute = attribute

def getatt_constructor(loader, node):
    return CFGetAtt(loader.construct_sequence(node))

def ref_constructor(loader, node):
    return CFReference(loader.construct_scalar(node))

yaml.add_constructor('!Ref', ref_constructor)
yaml.add_constructor('!GetAtt', getatt_constructor)

with open('../../small-test.yaml', 'r') as fh:
    pprint(yaml.load(fh))

Which returns

{'WebInstance': {'CreationPolicy': {'ResourceSignal': {'Region': <__main__.CFReference instance at 0x10b4426c8>,'Timeout': <__main__.CFGetAtt instance at 0x10b442710>}},'Type': 'AWS::EC2::Instance'}}

Python Docs

Python also has support for presenters, functions that take a class and return its Yaml equivalent for serialization. I'm not 100% how Ruby does this, perhaps by calling to_yaml on objects.

Unlike Ruby, Python will fail if it encounters a tag it can't construct. The Yaml spec says this about unresolved tags:

If a document contains unresolved tags, the YAML processor is unable to compose a complete representation graph. In such a case, the YAML processor may compose a partial representation, based on each node’s kind and allowing for non-specific tags.

A few of the features I think would be useful:

• Generalized tag constructors
  Python allows you to specify a prefix, and all tags starting with that prefix get passed to a specific constructor. This would be really useful for me, especially if that prefix can be blank (aka a catchall constructor). Alternatively, instead of registering constructors, this library exposes a single function that gets called for all unrecognized tags.
• Tags are dumped correctly
  This is really important for my use case, because I don't want to do anything with some tags. It'd be really nice if serialize(deserialize(yaml)) == yaml with respect to tagging. Or at least make this achievable (even if I have to translate tagged values to structs during deserialization, and when dumping those structs explicitly state what tag they should have when they're serialized).
• Unresolved tags
  I don't think unresolved tags should be an issue for this library. Unlike serde_yaml, this library doesn't deserialize to structs; so unresolved tags should be much less of an issue. Maybe unresolved tags get a special struct that derefs to the value (this way the tag is still available). This way serde_yaml could then fail if it encounters an unresolved tag, but this library wouldn't need to.

[DeltaEvo]

I need this functionality too , actualy I copy paste YamlLoader struct and I add this code in the on_event when event is Scalar in the match

I'ts a tag to include yaml tree into another

Exemple:

hello: !include 'hello.yaml'

The code

              if style == TScalarStyle::SingleQuoted {
                    if let Some(TokenType::Tag(ref handle, ref suffix)) = *tag {
                        if *handle == "!" && *suffix == "include" {
                            let mut content = String::new();
                            let node = match File::open(Path::new(self.root.as_str()).join(v)){
                                Ok(mut f) => {
                                    let _ = f.read_to_string(&mut content);
                                    match YamlLoader::load_from_str(content.as_str() , self.root.clone()) {
                                        Ok(mut docs) => docs.pop().unwrap(),
                                        Err(_) => Yaml::BadValue
                                    }
                                }
                                Err(_) => Yaml::BadValue
                            };
                            self.insert_new_node((node , aid));
                            return;
                        }
                    }
                }

But It will great to have an api for this (And more generaly an api to customise YamlLoader)

[jess-sol]

That's funny @DeltaEvolution, that was one of the use cases I wanted to add with tags, file inclusion. That's a decent example you have there for it, in the mean time.

[DeltaEvo]

@ioben ;) @chyh1990 If you want more exemple with another's libraries, I have the !include tag with snakeyaml in java

[DeltaEvo]

I have created a very simple implementation here

The parser with this api look like

    struct HelloParser;

    impl YamlScalarParser for HelloParser {
        fn parse_scalar(&self, tag: &TokenType, value: &String) -> Option<Yaml> {
            if let TokenType::Tag(ref handle, ref suffix) = *tag {
                if *handle == "!" && *suffix == "hello" {
                    return Some(Yaml::String("Hello ".to_string() + value))
                }
            }
            None
        }
    }

So you can parse not only tags but all types of scalar values

[flyx]

Hi folks!

Not sure how I ended up here, but since it has been asked how other libraries use this feature, I think I can contribute some thoughts. I am the author of NimYAML, which employs tags by default. Since most other YAML implementations are written for languages with dynamic typing, we might benefit from sharing ideas here.

First, a word on terminology: A tag directive is a prefix before the YAML document; what you are talking about is tag handles:

%YAML 1.2
%TAG !n! tag:yaml.org,2002: # << this is a tag directive
---
!n! tagged scalar # << this is a tag handle

Tag handles are used to explicitly denote the type of a YAML node. The type of a YAML node without a tag is deduced from its content and its ancestors in the same document. Tag handles can be employed for type safety. For example, in NimYAML, I can serialize a sequence of integer values like this (I hope this code is readable for people not knowing Nim; @[] is a sequence constructor):

import yaml.serialization
echo dump(@[1, 2, 3])

Which will generate:

%YAML 1.2
---                                                                                                                                                                                                               
!nim:system:seq(nim:system:int64) [1, 2, 3]

When loading this YAML file with NimYAML, it will check whether it is loaded into a sequence type that matches the tag of the value and raise an error if it doesn't. That way, type errors can be discovered early. Note that the numbers inside the sequence do not need tags because their tag is implicitly given by an ancestor (the sequence).

Now I understand that yaml-rust does not load YAML into native types, but rather into a DOM. This corresponds to the Representation (Node Graph) as defined in the spec. As you can see in the spec, tags are part of that Node Graph, so there is nothing yaml-rust itself is required to do with them if its API barrier is between the Node Graph and native data structures. Its only responsibility should be to collect the tags present in the YAML input and make them accessible in its DOM. NimYAML also has a DOM API, but its use is discouraged because serializing to and from native types is more convenient and automatically includes type checks.

A use-case which might be relevant for yaml-rust is tag handles in sequences or mappings with heterogeneous value types. An example from NimYAML documentation:

%YAML 1.2
---
- this is a string
- 42
- false
- !!str 23
- !nim:demo:Person {name: Trillian}
- !!null

To process this sequence, the type of each value must be evaluated. For scalar values, this could be done by guessing the type (this is a string -> string, 42 -> integer). But if you have multiple complex types that can occur inside this sequence, it can be very impractical (or downright impossible) to infer their type from the value. Therefore, using tags here is very helpful. Since I do not want to clutter this conversation with code too much, I will simply link to the NimYAML examples to show how this data is loaded with NimYAML.

So, to sum up: My advice would be to simply make tags queryable from your DOM interface and let the caller decide what to do with them. They are very relevant for implementing automatic (de-)serialization of native types, and perhaps you want to add that some day to yaml-rust.

Cheers!

[chyh1990]

Thanks for the detailed discussion!

Here's my options:

• Handling tags with hooks (in the same way in Python or Ruby) is easy, maybe it can be added in the next major release. @ioben @DeltaEvolution
• Unfortunately, yaml-rust does not implement a full DOM containing all metadata in the original text file (e.g. tags, mark, etc.) yet. This makes serialize(deserialize(yaml)) == yaml impossible. The serializer is preliminary, we need contribution for that. @ioben
• tag directives for standard types are already supported by yaml-rust (also type guessing), but for a strongly typed language like Rust, maybe it's not easy to do auto type casting? @flyx

@DeltaEvolution Can you generalize your patch and submit a PR?

[flyx]

@chyh1990 The root type must be known at compile time and it can only be checked whether the YAML has the correct tag on its root element for strongly typed languages, that is correct. However, if you have

enum Message {
    Quit,
    ChangeColor(i32, i32, i32),
    Move { x: i32, y: i32 },
    Write(String),
}

(forgive me, I do not really know Rust and just copy things from the documentation), you can use Vec<Message> as root type, which would make it possible to deserialize this to a native type:

- !Quit  # no value
- !ChangeColor [1, 2, 3] # sequence as child
- !Move {x: 23, y: 42] # mapping as child
- !Write some string # scalar as child

If you have exactly one value for each enum item, and all these values have different types, you can match the tag (or the guessed type) against the value types instead of using the enum items as tags. This makes it possible to deserialize complex heterogeneous structures. You do not lose type safety and do not do any type casting.

[chyh1990]

@DeltaEvolution propose a new API for tag parsing (see #37 ). It looks OK for me. Any opinion for the PR? @flyx @ioben

[trans]

I don't know Rust so forgive me that I can't give more Rust-oriented advice. But I know YAML quite well. What is often overlooked in implementations, beside the intermediate representation graph that @flyx mentions, are Tag Schema. I am not sure a single implementation of YAML currently in the wild actually handles this correctly. A complete implementation would allow for the definition of alternate schema in order to fully control the use of tags in the processes of loading and dumping, and of course include JSON and Core schema out of the box.

I am presently working on adding this support to Crystal, which is built on top of libyaml. When I am done I'll drop a line here if anyone wants to take a look for reference sake.

[flyx]

@trans I don't think that it's overlooked. The whole idea of having programming language independent tags never really took off, because YAML is being used more for serialization/deserialization within the same application and for configuration. In popular YAML implementations, you can specify custom tags in addition to those supported out of the box, and the spec advices that other schemas are based on the core schema, so I don't really see how that is not correct.

[softprops]

What's the status of this. I just surveyed the rust yaml ecosystem and this crate looks to be the most complete despite missing features like this that would make it complete or at least on par with other language ecosystems support for yaml tools.

Have others been able to work around this yet? I'm also needing to parse cloudformation yaml templates that leverages several !Foo tags https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html

[theblazehen]

https://github.com/chyh1990/yaml-rust/blob/9f7d84828b5d5468c47f9c104809a996c3f72c2e/tests/specexamples.rs.inc#L55-L73 seems like it's in the tests at least, and saw a few mentions in the code, hopefully it'll be released soon?

[softprops]

Is there an interface on master for registering tags?

[davvid]

FWIW I've merged #135 (a rebased version of #37) into my fork: https://github.com/davvid/yaml-rust