CJSONSerializer returns NSData that can't be converted to NSString

[atomicbird]

In some cases data returned from CJSONSerializer can't be converted to an NSString without losing some or all of the JSON data. I ran into this with the included test data file "twitter_public_timeline.json", which contains CJK characters. I deserialized the file using CJSONDeserializer and then re-serialized via CJSONSerializer. [Why? Because I have a slightly hacked CJSONSerializer that can produce more human-readable output. But the problem exists even on the stock CJSONSerializer.]

Somewhat simplified code (the full version is in https://github.com/atomicbird/jsonlint)

NSData *inputData = [NSData dataWithContentsOfFile:@"twitter_public_timeline.json"];
id json = [[CJSONDeserializer deserializer] deserialize:inputData error:nil]
NSData *jsonData = [[CJSONSerializer serializer] serializeObject:json error:nil];
NSString *jsonString = [[NSString alloc] initWithData:jsonData encoding:NSUTF8StringEncoding];

At this point jsonData contains what looks like good JSON, but jsonString is nil. If I write jsonData to a file it contains UTF-8 encodings of the CJK characters.

If I then run jsonData back into CJSONDeserializer, I get an error message reading "Could not scan data. Data wasn't encoded properly?", from CJSONScanner:

NSError *deserializeError = nil;
id json2 = [[CJSONDeserializer deserializer] deserialize:jsonData error:&deserializeError];

This looks to be because CJSONScanner is doing more or less the same thing I was doing above, attempting to create an NSString and specifying UTF8.

[atomicbird]

This code covers what I'm seeing. At the end, jsonString and json are nil, and the error message printed is the same as the one above:

{
    NSData *inputData = [NSData dataWithContentsOfFile:@"/...../twitter_public_timeline.json"];
    [inputData writeToFile:@"/tmp/input.json" atomically:YES];
    NSError *deserializeError = nil;
    id json = [[CJSONDeserializer deserializer] deserialize:inputData error:&deserializeError];
    if ((json == nil) || (deserializeError != nil)) {
        NSLog(@"Deserialize error: %@", [deserializeError localizedDescription]);
    }

    NSError *serializeError = nil;
    NSData *jsonData = [[CJSONSerializer serializer] serializeObject:json error:&serializeError];
    [jsonData writeToFile:@"/tmp/output.json" atomically:YES];

    NSString *jsonString = [[NSString alloc] initWithData:jsonData encoding:NSUTF8StringEncoding];
    NSLog(@"JSON string: %@", jsonString);

    deserializeError = nil;
    json = [[CJSONDeserializer deserializer] deserialize:jsonData error:&deserializeError];
    if ((json == nil) || (deserializeError != nil)) {
        NSLog(@"Deserialize error: %@", [deserializeError localizedDescription]);
    }
}

[schwa]

Closed by d0a78dee53770d185bdeaefc93c9ae9b5130f638 [someNSString length] != strlen([someNSString UTF8String).