Fails on very large input in chrome

[edalquist]

Error:

Uncaught SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at FileReader.r.onload (location.js:19)

This is on a 262MB JSON file.

[edalquist]

Works on Safari Version 10.0.2 (12602.3.12.0.1) Fails on Chrome Version 56.0.2924.87 (64-bit)

[AymenQ]

Same error here on Chrome 55.0.2883.87.

Firefox 50 shows "allocation size overflow" for location.js:18:12.

IE 11 isn't showing any errors (apart from some HTML warnings) but it still doesn't stop loading for some reason.

Don't have any other browsers to test. This is a 265 MB JSON file.

[AymenQ]

Seems that this is an issue with files larger than 256 MB. I don't think we can use JSON.parse, maybe JSONStream will work though.

[kinto0]

I've changed it from JSON.parse to jquery's $.parseJSON(). Could you guys check if it is working now? Thank you!

[edalquist]

Still no luck in Chrome:

Uncaught SyntaxError: Unexpected end of JSON input at Function.parse [as parseJSON] () at FileReader.r.onload (location.js:19)

[kinto0]

I've been trying to make my own JSON file that big but every time I copy it it crashes notepad++. Mine's 198 MB and I can't reproduce the error. If any of you can either send me your JSON (I personally wouldn't send this to a stranger on the internet) or find a fix for it that would be great. If not, we will have to just leave it as-is until my locationhistory exceeds the limit.

[edalquist]

Looks like V8 has a hard coded limit of 256MB for strings: https://github.com/google/trace-viewer/issues/627

[edalquist]

So I was able to get my history to work by simply stripping the whitespace using jq

jq -c . LocationHistory.json > LocationHistory.compact.json

That reduced my 263MB JSON file to 166MB. Maybe just catch the error and pop-up a link to download jq and instructions on how to use it to strip whitespace?

Eventually some people will hit the 256MB limit even after stripping whitespace. At that point jq could be used to replace all the "long" field names with single characters to get even more space.

[kinto0]

Thank you very much! I'll add the catch now. I was thinking a more future-proof solution for this would be to have them upload the zip instead because that is much much smaller. The only problem is I have no idea how to do that so it might take a while.

[edalquist]

Reading the V8 bug I think you'd hit the same problem. As soon as you have a string longer than 256MB you can't parse it into JSON.

[kinto0]

I've pushed the new update to the site. I've also trimmed the JSON before saving it to a variable so it should be a smaller size and might work without catching the error. Thanks for all the help!

[edalquist]

This isn't even as extreme as you could take it but it replaces all the keys with 2 character versions of themselves.

This takes my 263MB JSON file and strips it down to 112MB. If you also replaced the various activity type strings with single characters and figured out how to get jq to leave off the quotes from map keys you could probably get the string even smaller.

You'd need a translation layer in your app to convert back from the single character keys to the longer strings but that wouldn't be too hard.

convert_names.def:

def translate_key(from;to):
  if type == "object" then . as $in
     | reduce keys[] as $key
         ( {};
       . + { (if $key == from then to else $key end)
             : $in[$key] | translate_key(from;to) } )
  elif type == "array" then map( translate_key(from;to) )
  else .
  end;

.
    | translate_key("locations"; "l")
    | translate_key("accuracy"; "ac")
    | translate_key("latitudeE7"; "la")
    | translate_key("longitudeE7"; "lo")
    | translate_key("timestampMs"; "ti")
    | translate_key("activitys"; "ay")
    | translate_key("activities"; "ai")
    | translate_key("confidence"; "c")
    | translate_key("type"; "ty")

jq -c -f convert_names.def  LocationHistory.json > LocationHistory.supercompact.json

[AymenQ]

In order to fix this for good, I recommend using an input stream and parse it with JSONStream. I've seen this issue in other projects and it's been resolved by using JSONStream.

The only problem is that it's a nodejs module and you would need to use something like browserify. I would work on it but I haven't got much free time at the moment.

[XopherThomas]

I'm one of the lucky fools who have a very large Location History.json. I'm at 420MB. Using edalquist's suggestion above only cuts it down to 299MB. It would be great to see the project upgraded to support the larger location sizes. There are several create tools based on this that all have the same 256MB limit.