emitter does not support ScalarEvent with null tag

[GoogleCodeExporter]

A recent fix for the Psych YAML library (which wraps libyaml) started passing 
in null tags for scalar events during emitting. This appears to break in 
SnakeYAML, but work in libyaml.

Here's the JRuby code:

        ScalarEvent event = new ScalarEvent(
                anchor.isNil() ? null : anchor.asJavaString(),
                tag.isNil() ? null : tag.asJavaString(),
                new ImplicitTuple(plain.isTrue(),
                quoted.isTrue()),
                value.asJavaString(),
                NULL_MARK,
                NULL_MARK,
                SCALAR_STYLES[(int)style.convertToInteger().getLongValue()]);
        emit(context, event);

Note that anchor and tag can come in as "nil" from the Psych wrapper, in which 
case we make them null. In the libyaml version, the code looks like this (and 
works):

    yaml_scalar_event_initialize(
        &event,
        (yaml_char_t *)(NIL_P(anchor) ? NULL : StringValuePtr(anchor)),
        (yaml_char_t *)(NIL_P(tag) ? NULL : StringValuePtr(tag)),
        (yaml_char_t*)StringValuePtr(value),
        (int)RSTRING_LEN(value),
        plain ? 1 : 0,
        quoted ? 1 : 0,
        (yaml_scalar_style_t)NUM2INT(style)
        );

    emit(emitter, &event);

In both cases, "emit" just calls into the YAML library's emitter logic.

The JRuby version results in the following error:

  1) Error:
test_json_dump_exclude_tag(Psych::TestCoder):
RuntimeError: org.yaml.snakeyaml.emitter.EmitterException: tag is not specified
    org/jruby/ext/psych/PsychEmitter.java:161:in `scalar'

So it seems there's some incompatibility here with libyaml.

Original issue reported on code.google.com by head...@headius.com on 13 Sep 2011 at 6:29

[GoogleCodeExporter]

JRuby bug: http://jira.codehaus.org/browse/JRUBY-6067

Original comment by head...@headius.com on 13 Sep 2011 at 6:31

[GoogleCodeExporter]

(a little bit off topic..)
1) SnakeYAML does not claim it is compatible with libyaml because it is not. It 
is only fully compatible with PyYAML. Even though libyaml and PyYAML are from 
the same developer and they are very similar, they still deviate. 

2) I am impressed with the way you use SnakeYAML. And I must admit I do not 
understand it. In the version 1.9, SnakeYAML has a number of methods to work 
with low-level API. (parse, compose etc). But there is no way to emit events. 
This is not because I want to hide anything but because I do not know how to 
implement it. The way to pass the 'context' for the emitter is so ugly that I 
did not dare to put it into the public API. Apparently, you have solved the 
problem.
Can you may be share with us the way how you emit events ? May be we can put it 
to our public API. (explanation, description, code - anything)

(back to the point)

3)
Events are created out of tokens when the document is parsed or out of 
representation tree (node).
When the document is parsed then this event
ScalarEvent(anchor=null, tag=null, implicit=[true, false], value=5)
means:
there is no explicit tag and the value '5' is a subject for the implicit tag 
recognition (via a set of regular expressions)
Normally, it would mean that '5' will become an Integer. But SnakeYAML may 
create a Long (when Integer does not fit) or String (if it is a value for a 
JavaBean property of type String)

While when an object is emitted then the situation is different. Without a tag 
there is no way for the emitter to find out how to emit '7'. Integer and String 
must be emitted differently. It makes it even more complicated when the 
emitter's 'context' dictates single or double quoted values (because implicit 
tags are gone). 
To give you more information I have created a test:
http://code.google.com/p/snakeyaml/source/browse/src/test/java/org/yaml/snakeyam
l/issues/issue132/ScalarEventTagTest.java
Feel free to take it and play with it.

It means the following. Two bugs are combined together in a way that it works 
under curtain conditions. First bug is in Psych, it loses the runtime 
information about a value (integers become strings). Second bug is in libyaml, 
it makes an assumption that if a tag is not present it should be string.
When you only emit strings then these 2 bugs work together like a charm :)

We can continue as following:
a) play with JRuby to prove that integers (dates, booleans and others 
http://www.yaml.org/type/) are not emitted properly
b) file a bug in Psych that the tag is required
c) [optional] file a bug in libyaml that it should not make an intelligent 
guess about the tag when it is not explicitly provided for emitter (because it 
is not intelligent enough :)
d) share with us the results of your investigation

Feel free to ask further questions.
Andrey

Original comment by py4fun@gmail.com on 14 Sep 2011 at 9:13

• Changed state: New

[GoogleCodeExporter]

This can be closed as wontfix for now.

The author of the "psych" library we emulate atop SnakeYAML has been updating 
the general structure of psych and our specific implementation to pass more 
tests. He has also decided he likes how SnakeYAML is structured and is moving 
psych toward that structure.

This issue was reported by me primarily because I saw it in psych tests, but 
with the original author attempting to clean those up we can ignore this for 
now.

Original comment by head...@headius.com on 28 Sep 2012 at 9:32

[GoogleCodeExporter]

Original comment by py4fun@gmail.com on 28 Sep 2012 at 10:38

• Changed state: WontFix