request for mochijson2 to reject string input that is not a valid unicode encoding

[GoogleCodeExporter]

What steps will reproduce the problem?
1. mochijson2:decode(<<34, 128, 34>>). % this succeeds

2. mochijson2:encode(mochijson2:decode(<<34, 128, 34>>)). % this roundtrip fails

What is the expected output? What do you see instead?
I'd like to see mochijson2 reject this input as invalid.  Instead it decodes 
this as <<128>>, which 
fails the roundtrip:

mochijson2:encode(mochijson2:decode(<<34, 128, 34>>)).
** exception exit: {ucs,{bad_utf8_character_code}}
     in function  xmerl_ucs:from_utf8/1
     in call from mochijson2:json_encode_string/2

What version of the product are you using? On what operating system?
r97, any OS.

Discussed in detail in https://issues.apache.org/jira/browse/COUCHDB-345.  I 
think the ideal 
solution from CouchDB's perspective would be for mochijson2 to accept input in 
any Unicode 
encoding.  An interim solution that we've adopted is to reject anything other 
than UTF-8.  I'm not 
sure if that's an appropriate solution for the broader Mochiweb community, but 
nevertheless I'm 
attaching the patch.

Original issue reported on code.google.com by adam.kocoloski@gmail.com on 10 Nov 2009 at 3:05

Attachments:

• mochijson2_input_validation.diff

[GoogleCodeExporter]

mochijson2 is intended to work ONLY with UTF-8 input and output, so this isn't 
a problem. I'll try and take a 
closer look at the patch later this week, it would be better if there were 
tests! :)

Original comment by bob.ippo...@gmail.com on 10 Nov 2009 at 3:12

[GoogleCodeExporter]

Thanks for the nudge, I'll see if we can get some tests attached to that patch.

Original comment by adam.kocoloski@gmail.com on 10 Nov 2009 at 3:37

[GoogleCodeExporter]

Here's an updated patch with some simple tests for invalid 2/3/4-byte sequences

Original comment by adam.kocoloski@gmail.com on 11 Nov 2009 at 2:50

Attachments:

• mochijson2_input_validation_with_tests.diff

[GoogleCodeExporter]

Applied in r112

Original comment by bob.ippo...@gmail.com on 15 Nov 2009 at 9:53

• Changed state: Fixed