fwkoch
commented
5 years ago Alternative Solution: Standard zip format
Response to pros/cons
Writing as pure JSON is impractical for large file sizes. Regarding the "pros" above -
- human-readable - many text editors cannot open large (~ GBs) files. Some can (like
less), but easy access by all users in unrealistic. - easy validation - in-line base64 binary does not help with this, binary blobs are relatively opaque, only easy access to metadata is useful for validation.
- standard format - This is valid and solved by pure JSON but could also be solved by other standard formats.
Regarding the "cons" -
- The requirement to load the entire file into memory is a significant issue. At best, this means all operations are slower, since every time we need to read or write individual objects, we must load the file. This is a regression from v1 of OMF which allows reading only lightweight metadata then stepping directly into the binary data only when required. At worst, large objects (e.g. block models with many attributes) become unsupported simply because of the chosen file format. The implications of this "con" should not be trivialized.
Proposed Solution: Zip archive
Store JSON metadata and binary as separate files within a standard, zipped folder structure. The header/metadata would look similar to the metadata above, but instead of base64 inline, binary is specified by paths to files. Something like
my_file.omf
|- metadata.json
|- pointset_vertices_array.bin
|- surface_data/
| |- vertices.bin
| |- triangles.bin
| |- data.bin
| |- image.png
|- another_folder/
|- possibly_more_metadata.json
|- other_binary_file.binPros
- Standard format, easily explored by all users
- For zipping, there are several options, but all are open and well supported (
zip,7zip,tar.gz). By changing the file extension from.omfto.zipstandard OS file browsers can open and view files. - Then, all the included files are also standard. Metadata is pure JSON (but in much smaller, manageable files with no binary). Other binary data can be either known format (e.g.
.png) or simple binary array with type info still stored in metadata.
- For zipping, there are several options, but all are open and well supported (
- Easy to use without reading everything into memory. Files can simply be added to the folder structure, then only metadata needs editing.
- Still looks like a single, simple
.omffile to users who are only concerned with the full file, not internal data.
Cons
- Including unrelated, unwanted binary becomes very easy - just unzip, add the binary, re-zip. However, in theory, unwanted binary could be included in the other formats as part of the binary blobs. In fact, by exposing a file system, exactly what is saved in the file becomes more explicit, so possibly this is an advantage?
rowanc1
TroyWilliams3687
dbrookes96
rpakdel
hishnash
Zekaric
martinken
exepulveda
Context
The current format
OMF-v1starts with a 60 byte header, followed by a JSON dictionary containing all elements of the project keyed by UID strings. This JSON is followed by a series of individually zipped binary blobs. Objects can reference each other by UID, and arrays and images contain pointers to their data in the binary blob.See the documentation here: https://omf.readthedocs.io/en/latest/content/io.html#omf.fileio.OMFWriter
The current format allows you to read the JSON data from disk, without loading all data items.
Problem
The current serialization format is not human readable out of the box, and requires a custom implementation to both read and write; this increases the barrier to entry for adopting OMF. Moving away from a custom binary format would simplify reading the file and make it easier to validate in multiple languages.
Proposed Solution
Move to Base64 encoding for the data-arrays and serve the entire file structure as JSON. This would keep the general structure of the binary data as individual resources.
Pros
Cons
Overlap with other issues
colorto metadata****Geometryand flatten this to be on elements.__class__totype#42Example