Octopus 3 compression techniques.

[scoiatael]

In GitLab by @nikitamironov on Jan 14, 2020, 11:03

Problem

Octopus can represent large amount of data. We can try to compress it in different ways.

Possible solutions

There are different ways to achieve the goal.

Binary format

There are a lot of different binary formats we can use, such as protobuf, flatbuffers, bson...
Pros:

• dramatically decreased size of data
• strict typing of data
• custom structures

Cons:

• files will require additional libraries to parse and work with

  Custom binary format

  We can invent our own binary format with less abstract or less advanced specification.
  Pros:

• same as other binary formats
• ability to customize
• more simple (or more advanced, depends on us) than public opensource solutions

Cons:

• requirement to maintain custom libraries, testing and so on
• more closed to others, difficult to work with without any previous consultation with us

JSON with more compact data representation

In previous versions of Octopus we didnt care much about size. As a consequence we have quite big result JSON file with different kinds of data. Optimizations can help us save some bytes by restructuring some data, renaming its properties and other ways.
Important thing we can not forget about is readability of file. Of course its possible to write only and only data without any descriptive keys and structures, but it will be pain to work with such data files.
We can use the following techniques, its topic to discuss:

• Excluding shape descriptor and work with it as with any other bitmap asset. We will have more files, but one shape file can theoretically be referred multiple times by different layers. Also, Octopus itself will become more lightweight. There are pros and cons of it too. Personally I think its not that necessary for now.
• Usage of linkings. We can exclude different kinds of repetitive descriptors and put it into some root property, such as assets. Later we can just link same kind of information anywhere in document without complete copying. Pros are high volume of saved bytes, cons are that layers won't be such independent instances and there will be requirement to either join data from assets into given layer or provide global assets object to work with given layer.
• Short key names. There is no reason to use long names of keys, even lineCap can be replaced just to cap and world will spin as before. Its quite good technique which will cost us nothing but will reduce size a lot.
• Removing of obsolete or redundant data. Some of data in Octopus are totally useless for us atm, so we can just remove it.

[scoiatael]

In GitLab by @nikitamironov on Jan 14, 2020, 11:05

Personally I'd choose JSON with compact representation. I still think about linkings, it will really reduce size of file but its quite sensitive thing, I am not sure we are 100% ready for it. Smaller key names and removed redundant keys are must-have.

[scoiatael]

In GitLab by @janroztocil on Jan 14, 2020, 13:28

In general, I'm not against a reduction of Octopus byte size, however, I'd prefer to address it as an additional step once the new model and its working implementation are ready. The main reasons for JSON format are readability for us and for the third parties, low complexity and no necessity for external dependencies. Also, this will establish a baseline we can later use to benchmark an eventual optimization.

This considered Octopus as a whole, we might, however, apply local optimization where it makes sense, namely to the shape geometry representation. There was a proposal for having atomic paths encoded using SVG Path notation, this seems like a good idea, it's compact and yet universally understood format.

Regarding the offloading of shape geometry into assets, I'm not particularly keen on that. The complexity of that this feature is not imo justified enough.

And on shortening the keys... eh, why not, it won't do any noticeable difference, but there's no reason for having them unnecessarily long.