search

opentoonz / opentoonz

OpenToonz - An open-source full-featured 2D animation creation software
https://opentoonz.github.io/
Other
4.53k stars 525 forks source link

[Clean/Rewrite] A rewrite of tiio_tzl (Write/Read of levels/drawing *.tlv files) #3739

Open le-dragon-dev opened 3 years ago

le-dragon-dev commented 3 years ago

Current state

Currently, levels and drawings are saved into a *.tlv file, thanks to the file tiio_tzl.h/cpp. It is a really important file! So, why should we change something if it works?

The problem

I start to add a unit test system for OpenToonz and I look around files to find some MVP tests to implement for a first pass, and I because the read/write of tlv file is important, I wan thinking is a good way to start! But...

The tiio_tzl.cpp file has 2473 lines, written in "C++" and C, with some unused code, weird comments or comment in Italian (not useful for an international project), retro-compatibility with older Toonz version (even before OpenToonz was born), etc...

Even if it works, it can be a nightmare to read, debug, fix, extend, or just know how it works.

The idea

The idea is to keep strictly the same architecture of *.tlv, but rewrite the reader/writer cleaner, better, stronger. Make a documentation to explain how it works and let nice comment for next developers.

What we will gain as a developer

What we will lose as a developer

What we will gain as a user

What we will lose as a user

Feel free to add comment and/or feelings about it!

RodneyBaker commented 3 years ago

Firstly I want to thank you for taking on this task. You know I'm all for it (as discussed previously via discord).

If I can cherry pick one thing you mentioned...

comment in Italian (not useful for an international project)

I have found the commentary in three primary languages of Italian, Japanese and English to be quite useful in that it reveals the vintage of the associated code at a glance. Specifically, although there are some exceptions, Italian is the oldest code with most of that sourced from Digital Video... Japanese commentary is of Studio Ghibli vintange and English is largely from the point of open sourcing of Opentoonz code moving forward.

For what it is worth, the different languages for me has been very useful in understanding the context in which the code was written. It is fairly trivial to translate the comments via google translate for developers who need to dive into the code and the commentary does get updated (to English) when the older code is replaced.

le-dragon-dev commented 3 years ago

TLV file structure, let's go!

Those files are in little endian This is the file structure of the version 14 of TZL/TLV files

TLV file header

Name Type Offset Size Description
Magic word String 0x00 8 bytes A simple world to be sure it is a TLV file, with the version number. Currently, only the first 5 bytes are tested.
Creator String 0x08 40 bytes The creator is currently the name of the application, its version and a system of compatibility mask (CM). You can check the file txshsimplelevel.cpp for more information.
HDR size TINT32 0x30 4 bytes
LX (Width) TINT32 0x34 4 bytes Width of the level
LY (Height) TINT32 0x38 4 bytes Height of the level
Frame count TINT32 0x38 4 bytes Number of frames in the level
Offset table position (Frame) TINT32 0x3C 4 bytes Information about each frame: ID(number/letter), offset in file and data size
Offset table position (Icon) TINT32 0x40 4 bytes Information about each icon: ID(number/letter), offset in file and data size
Codec String 0x44 4 bytes Name of the codec to use to read the data (LZ0 compress)

Icons are in fact thumbnails used for each drawing/frame

TLV file offset table (frame/icon)

This table will apply for each frame/icon (loop). (First offset start at "Frame/Icon" Offset table position)

Name Type Offset Size Description
Frame number TINT32 0x00 4 bytes Number of the frame/icon
Frame letter char 0x04 1 byte Letter of the frame/icon
Frame offset TINT32 0x05 4 bytes Offset of the frame/icon (where its data start in the file)
Frame size TINT32 0x09 4 bytes Size of the frame/icon data

TLV file frame/icon data

Frame are saved using a "Save box": Only save the part of the level with data. Name Type Offset Size Description
SaveBox x0 TINT32 0x00 4 bytes Top-left x axis of the saving box
SaveBox y0 TINT32 0x04 4 bytes Top-left y axis of the saving box
SaveBox width TINT32 0x08 4 bytes Width of the saving box
SaveBox height TINT32 0x0C 4 bytes Height of the saving box
Pixels data size TINT32 0x10 4 bytes Pixel data size (compressed LZ0!)
DPI x double 0x14 8 bytes Dot per inch on x axis
DPI y double 0x1C 8 bytes Dot per inch on y axis
Pixels data char 0x24 "Pixels data size" bytes Compressed pixel data

TLV file pixel format "CM" (TPixelCM32)

CM stands for ColorMap, it was introduce in Toonz 5.0 and its size is 32bits:

WolfInABowl commented 3 years ago

8 bits for tone (alpha?) I believe the Tone value is used to linearly interpolate between the Ink and Paint colors. This allows for creating the appearance of more colors than the two that are actually assigned to any given pixel.

This results in the ability to create soft-edged lines, as well as preserving correct pixel color when converting monochrome images to TLV. It is the reason why it is possible to fill soft-edged shapes without any issue. Filling simply replaces the Paint index, so instead of interpolating to a fully transparent style, the pixel instead interpolates to the fill color.

Here's an example use-case - converting a scanned black and white pencil drawing to a TLV: The brightness of each pixel is used to determine the Tone value, with brighter pixels resulting in higher Tone values. By default, each pixel is assigned a fully transparent Style (0) and a fully opaque black Style (1). This means that the brighter pixels will be more transparent and the darker pixels will be more opaque. This produces a result where only the pencil lines are visible, with the white of the paper removed. Additionally, due to the 255 potential Tone values, the fine details of the lines are well preserved.