timo / json_fast

a naive imperative json parser in perl6, to evaluate performance against JSON::Tiny
Artistic License 2.0
27 stars 20 forks source link

Could JSON::Fast support multiple JSON objects? #54

Closed CurtTilmes closed 5 years ago

CurtTilmes commented 5 years ago

I realize this is my problem, not yours, but an external provider who is supposed to send JSON is actually sending me multiple JSON objects, Think something like this:

{ "a": 1 }{ "b": 2 }{ "c": 3 }

JSON::Fast correctly throws an Exception because of the extra garbage after a successful parse.

I am trying to figure out the right approach to solve my problem. Though I could fork JSON::Fast and make a JSON::Fast::Multi or some such, that would give me all sorts of other problems, keeping things in sync, etc.

I am wondering if you would be open to one of two approaches:

1) Add a :multi option to from-json() that instead of dying would just keep parsing and return all the objects in an array, something like this:

our sub from-json(Str() $text, Bool :$multi) is export {
    my str $ntext = $text;
    my int $length = $text.chars;
    my int $pos = 0;
    my @results;
    loop {
        my $result = parse-thing($text, $pos);
        try nom-ws($text, $pos);
        if $multi {
            @results.push($result);
            if $pos == nqp::chars($text) {
                return @results.elems == 1 ?? @results[0] !! @results;
            }
        }
        else {
            return $result if $pos == nqp::chars($text);
            die "additional text after the end of the document: { substr($text, $pos).perl }";
        }
    }
}

2) Leave everything exactly as it is, but add an extra from-json-multi() function something like this:

our sub from-json-multi(Str() $text) is export {
    my str $ntext = $text;
    my int $length = $text.chars;
    my int $pos = 0;
    my @results;
    repeat {
        @results.push: parse-thing($text, $pos);
        try nom-ws($text, $pos);
    } until $pos == nqp::chars($text);
    return @results.elems == 1 ?? @results[0] !! @results;
}
timo commented 5 years ago

I've finally put in a solution for this, but it follows neither approach. You can now catch an exception called X::JSON::AdditionalContent, which gives you the parse result, and tells you where in the source string you can resume parsing for the next one. hope that helps! otherwise, feel free to re-open this issue