JSONTokenizer doesn't recognize all instances of Non-breaking spaces

[GoogleCodeExporter]

Pasting into a JSON file a long string generated from the Output panel in
the Flash IDE will generate whitespaces that are not recognized by the
JSONTokenizer class. Trying to parse it will generate an exception:

[Fault] exception, information=JSONParseError: Unexpected character ' '
(code=160) encountered
Fault, parseError() at JSONTokenizer.as:544
 xxx            throw new JSONParseError( message, loc, jsonString );

(Note: The extra information with the charcode was added to the original code)

To fix this the following function in JSONTokenizer:

private function isWhiteSpace( ch:String ):Boolean
{
    return ( ch == ' ' || ch == '\t' || ch == '\n' || ch == '\r' );
}

can be replaced with:

private function isWhiteSpace( ch:String ):Boolean 
{
    return ( ch == ' ' || ch == '\t' || ch == '\n' || ch == '\r' ||
ch.charCodeAt(0)==160 );
}

Original issue reported on code.google.com by oma...@gmail.com on 29 Jul 2009 at 10:19

[GoogleCodeExporter]

I don't think this is a bug in the JSON parser in strict mode.  According to 
the RFC, whitespace is defined as:

ws = *(
                %x20 /              ; Space
                %x09 /              ; Horizontal tab
                %x0A /              ; Line feed or New line
                %x0D                ; Carriage return
            )

Character 160 is a non-breaking space, and not valid JSON whitespace.

However, if the parser is used in non-strict mode then I can deal with this 
better by including the non-
breaking space as whitespace.

Original comment by darron.schall on 2 Aug 2009 at 8:10

• Changed state: Accepted
• Added labels: strict

[GoogleCodeExporter]

Fixed in r93

Original comment by darron.schall on 3 Aug 2009 at 12:18

• Changed state: Fixed