Should cloudmc use downloadDataFile to return json?

[robnagler]

This code is what I'm concerned about:

            reader = vtk.IO.Core.vtkHttpDataSetReader.newInstance();
            cacheReadersByVol[volId] = reader;
            res = reader.setUrl(volumeURL(volId), {
                compression: 'zip',
                fullpath: true,
                loadData: true,
            });

• What is doing the zip compression?
• Should zip be gz instead? (more efficient, and what we normally do)
• The data structures returned are large. It would be more efficient to use msgpack. This bypasses websockets. Perhaps this should be an analysisJob?
• It looks like the data is being saved is our own format. It's being saved as json, and the files are quite large:

  42M ./e6UiDd1H/cloudmc/7eBUwhbG/openmcAnimation/outlines.json
  42M e6UiDd1H/cloudmc/7eBUwhbG/openmcAnimation/outlines.json
  23M ./e6UiDd1H/cloudmc/OWpTUJwJ/openmcAnimation/outlines.json
  23M e6UiDd1H/cloudmc/OWpTUJwJ/openmcAnimation/outlines.json

  JSON is not really a good format for megabytes of data. Is there a better format?

[mkeilman]

We do the compression - I believe we need the structure of the zip file that gzip lacks, but we can experiment. vtkHttpDataSetReader does support gzip.

json is convenient but another format could work. It can be completely numeric - as long as volume ids are always numbers

[robnagler]

I believe we need the structure of the zip file that gzip lacks, but we can experiment.

I see.

json is convenient but another format could work. It can be completely numeric - as long as volume ids are always numbers

Can vtkHttpDataSetReader accept an Object with the data, i.e. data that was received elsewhere?

[moellep]

vtk.js has a special web-only format which vtkHttpDataSetReader uses to read full volumes (polys and points and other data) all in one zip file. If it isn't provided as a zip file, I believe it makes many requests for each component. The vtkHttpDataSetReader.setURL API makes it really convenient to load a volume - no extra processing is required.

The outlines.json problem is unrelated - I've opened a new issue for that #7332