Allow for adjusting decimal precision on export in editor

[sterlingcrispin]

hey it would be great to have an option to reduce the floating point precision when exporting things from the threejs editor, I uploaded a file with two decimal places and after exporting it had 14-15 decimal places. For objects with 1m+ vertices that can add up quickly

[mrdoob]

Yes! The problem is that I don't know how to do it. JSON.stringify() doesn't have an option for that. The only option I can think is to run a regex on the output...

[mrdoob]

Oh wait. http://stackoverflow.com/a/9340239/1137134

[sterlingcrispin]

Also somehow related, the editor can import files that its unable to export due to stringfy running out of "memory to allocate" even on a system with loads of free memory , I'll submit a new ticket with more details if needed On Mar 10, 2015 3:02 AM, "Mr.doob" notifications@github.com wrote:

Oh wait. http://stackoverflow.com/a/9340239/1137134

— Reply to this email directly or view it on GitHub https://github.com/mrdoob/three.js/issues/6206#issuecomment-78024695.

[bsenftner]

Years ago I was a maintainer of an open source 3DSMAX Ascii Exporter. At that time when facing a similar question of what precision to use for floating value outputs, we found allowing for the number of decimal places to be specified at export introduced cracking in exported models.

We tracked that down to two issues: 1) many of the common implementations of float to string have a bug where garbage bits are added at high precision decimal places if the requested precision is too high, and 2) determining the precision necessary for a given model is not simple, and the logic required is questionable in an exporter.

The solution we found was to first test a float to be exported for requiring precision bits, if needed that export used the “%g” (scientific notation) formatter in the exporter’s C code handling of it’s export.

Using scientific notation for precision bit requiring floating outputs insures no more digits are output than necessary, while insuring the necessary precision bits that would prevent mesh-polygon cracking are exported.

On Mar 10, 2015, at 9:12 AM, Sterling Crispin notifications@github.com wrote:

Also somehow related, the editor can import files that its unable to export due to stringfy running out of "memory to allocate" even on a system with loads of free memory , I'll submit a new ticket with more details if needed On Mar 10, 2015 3:02 AM, "Mr.doob" notifications@github.com wrote:

Oh wait. http://stackoverflow.com/a/9340239/1137134

— Reply to this email directly or view it on GitHub https://github.com/mrdoob/three.js/issues/6206#issuecomment-78024695.

— Reply to this email directly or view it on GitHub https://github.com/mrdoob/three.js/issues/6206#issuecomment-78087513.

[makc]

This is same as #6132, no? @bsenftner my wild guess is that you had cracks in T-shaped junctions, in which case it is actually model problem (it always had cracks, but moving the vertices - which is essentially what happens when you alter the numbers - made them visible).

[makc]

@mrdoob that's cool compact repacer, but I think you need toPrecision. I initially thought toFixed as well, but it sometimes explodes the digits or simply kills the number, for example (1e10).toFixed(3) -> "10000000000.000" (1e-10).toFixed(2) -> "0.00"

[sterlingcrispin]

hey @makc yes #6132 seems like the same suggestion but I didnt find your issue when I searched for "decimal" because you used the word "digit" and included a screenshot rather than a search friendly explanation of the issue. Also I agree about your analysis regarding @bsenftner 's T-shaped crack issue,

I don't think calculating the precision necessary is a good idea, just offering a popup dialogue allowing users to select is best (Rhino does this).

In the short term I wrote these rough python scripts that read through threejs scene .JSON files and reduce the decimal count, and remove indentation (for file size sake). Feel free to use or improve them https://github.com/sterlingcrispin/public/tree/master/threejs

I don't know the best way to solve this in javascript but this short term fix is fine for my use. If the scale of your model is over 100 then a random rounding error of 0.01 is imperceptible

Also - these python scripts mentioned above managed to reduce a 161MB JSON file to 74MB and then when they were uploaded to threejs.org/editor chrome was using 33% less system memory (332mb rather than 493mb) .

[makc]

imagine how much less would the files be if 3js was using binary files instead ;) e.g. 4 bytes per number, and no 'decimal places' problem (json's 4 bytes are "1.2,")

[mrdoob]

imagine how much less would the files be if 3js was using binary files instead ;) e.g. 4 bytes per number, and no 'decimal places' problem (json's 4 bytes are "1.2,")

Soon soon! :)

[makc]

Just in case you implement the replacer, do this:

function (key, value) {
    return (value.toPrecision && (Math.floor (value) != value)) ? Number (value.toPrecision (p)) : value;
};

or else indices are screwed.

[makc]

Soon soon! :)

why wait? actually disregard that, it's ineffective, I tested and it turns 5.5MB eiffel json into Uint8Array of length 5063157.

[makc]

well I tried to hack quick binary i/o myself here, and it does not seem to be much more effective than json with truncated numbers. e g my 5.5MB BufferGeometry json goes down to 1.2MB when converting to indexed BufferGeometry with precision replacer above, and only to 963KB when converting to binary. so the gain is not that impressive (and gzip transfer encoding is probably making it even less so).

[makc]

well, maybe you could cut the number of bytes for floats from 4 to 2, using Uint16Array and bytes voodoo I can see that easily decreasing the size by another 40% to maybe somewhere around 500K. still not quite good as 106KB original .max file, but

I shall try at some point later and report the effects.

[antont]

we tested compressed binary with the Open3DGC supported by glTF (at least back then), our test scene got down from 22MB to 5MB or so iirc .. https://github.com/KhronosGroup/glTF/wiki/Open-3D-Graphics-Compression.

and that was compared to uncompressed binary, as ascii JSON i think that scene was / would have been many tens of megs.

[repsac]

I tried msgpack compression and it did help bring down the size a bit, but not on animation exports. I think because msgpack is only effective on floats and there is a lot of verbose string data in the animation format.

[crobi]

Since people were mentioning binary formats, I want to share my experience:

My COLLADA converter converts COLLADA files to either the three.js JSON format or a custom format similar to glTF (JSON/binary combo). The following are the file sizes for an animated character (2052 vertices, 3629 triangles, 70 bones, 464 keyframes):

• COLLADA: 9.9MB
• three.js JSON (text, all numbers quantized to 6 digits): 2.0MB
• custom format (binary float32, no compression): 760kb binary, 25kb JSON

The biggest chunk of the data for this character are animations. Since my converter is resampling animations, the by far biggest factor in the file size is the sampling rate. Going from 10fps to 30fps almost triples the file size.

Also, one more thing about binary formats: if you load binary data via XHttpRequest with responseType = 'arraybuffer', you can pass it more or less without any preprocessing to the GPU (just create several ArrayBufferViews from the big blob you received). Of course, you give up on fancy compression algorithms like the one @antont mentioned, but it lets you load models of arbitrary size in about a millisecond (once their download is finished). So if loading speed is more important than download speed to you (e.g., because you use a cache or work with local files), this might be something to consider.

[mrdoob]

Just as an update, I'm trying limiting the float precision to 6: https://github.com/mrdoob/three.js/commit/c9dd661e5fa2c00357d1f9bb34c1b6230b1215f7