Open sunipkm opened 4 months ago
A substantial portion of this crate is devoted to encoding and decoding image file formats. I don't quite understand the benefit of inventing our own bespoke image format to serialize and deserialize DynamicImage
s when users can instead use one of the standard formats
The encoding/decoding methods are not ubiquitous and require further setup of the encoders and decoders. Serialization and deserialization are not meant for transporting image data across different devices and architectures, instead for ease of programming where, e.g., I want to capture image data and create a DynamicImage
, serialize it, send it over IP to a client, and unpack it into a DynamicImage
, and continue processing it.
I suppose I should link this related PR which stalled out, but was trying to add uncompressed serialization/deserialization support.
If we do add serialization, I think it is very likely that people will use it to send images between different devices are architectures. But that's not necessarily a problem if we design it well. And there is a lot to be said for the convenience argument of not having to manually go through the process of encoding and decoding, particularly if the DynamicImage
is within a larger structure.
Perhaps the thing to do would be to lean on our existing codecs? Have the wire format actually look like DynamicImage(Vec<u8>)
and the blob of bytes be a PNG or TIFF encoded blob. That would likely get us better compression ratios and faster performance over using flate2 directly. And if we used standard image formats we wouldn't have to worry about versioning or mismatches between different image
releases
Using PNG/TIFF is fine as long as the data can be transferred without loss (including channel information etc). However, in my understanding, serialization-deserialization is opaque (correct me if I am wrong). As long as care is taken to account for multi-byte-endianness (which I do), I don't see why the system has to marry an image format.
Any way of serializing image data is essentially by definition an image format. The only question is whether it is a bespoke format that we've created ourselves or something standardized. Part of the benefit of picking a standard format is that we don't have to design one ourselves. But there's also benefits in terms of making sure that the format isn't accidentally changed between versions of this library (people may serialize with one version and deserialize with the next) and the higher level of testing and optimization we've already done for our existing formats.
I see how we can benefit by picking a standard format. As long as PNG (or some other, standard format) can accept all the variants of DynamicImage
and transfer them in a completely lossless manner, then standard format is definitely the way to go. But, I think, none of the encoders support F32 images (correct me if I am wrong). Hence, the 'simple' serialization-deserialization format where the data is compressed and encoded, with ancillary information (width, height, color type, byte endianness).
The TIFF format supports floating point. If the encoder doesn't currently allow it, it shouldn't be too much work to add
Will take a crack at it.
Adding 32F support was straightforward, but tiff encoder does not support gray images with alpha channel.
The PNG encoder should support all the integer formats (and has generally been better optimized compared to the TIFF encoder). The magic bytes at the start of the file will indicate which format it is
PNG for integer images, TIFF for floating point, then.
On a client-server type application where the server can attach to a multitude of sources of image data, a
DynamicImage
is the ideal structure to encapsulate the image data. However, sinceDynamicImage
does not support serialization-deserialization, sending the data from one process to the other is not trivial and requires conversion to custom, intermediate formats.I have attached an implementation: