search

petrbroz / svf-utils

Utilities for working with the SVF file format used by Autodesk Platform Services.
https://petrbroz.github.io/svf-utils/
MIT License
123 stars 53 forks source link

Add option to save glTF as msgpack format (or BSON) from the commandline and avoid crashing from JSON.stringify(this.manifest.. #55

Closed wallabyway closed 2 years ago

wallabyway commented 2 years ago

in the src/common/gltf/writer.ts file, this line 119...

    fse.writeFileSync(gltfPath, JSON.stringify(this.manifest, null, 4));

... will crash JSON.stringify, because the this.manifest object is too large to serialize into a string (nodejs 8GB option).

Instead, allow the writer to generate a msgpack (or BSON) output file, without invoking the JSON.stringify().

The current branch experiment/BSON uses a postProcess, but still executes JSON.stringify, and hence still crashes.

In my limited testing, exporting to msgpack does the following:

  1. avoids crashing
  2. the .msgpack file is significantly smaller than the json version
  3. gzipping the .msgpack is slightly smaller than the gzipped json file
  4. generating a msgpack output file, is significantly faster than outputting a json file
  5. decoding a msgpack gltf file, is about the same speed as a json gltf decoder.

note: I used the faster... const msgpack = require('msgpackr'); //https://github.com/kriszyp/msgpackr and not const msgpack = require('msgpack'); https://github.com/petrbroz/forge-convert-utils/blob/experiment/bson/samples/output-bson-msgpack.js

petrbroz commented 2 years ago

In https://github.com/petrbroz/forge-convert-utils/commit/b3397ebbccee75ccfff95cce7e7b1f49e054ad6c I've updated the GltfWriter class so that you can override the actual serialization process. And the https://github.com/petrbroz/forge-convert-utils/blob/develop/samples/serialize-msgpack.js sample shows how you can pack the in-memory glTF manifest directly to msgpack.

wallabyway commented 2 years ago

related to this.. https://github.com/KhronosGroup/glTF/issues/1699