Proposal: Optimize Palettes and Enhance Alpha

[phoddie]

Following the release of QOI, we have been discussing the possibility of whether to support a new lossless image format to the Moddable SDK. QOI is impractical because of the lack of RGB565 support. SLIC supports an RGB565 profile, which led to an exploration of potential applications of SLIC.

Our integration prototyping effort found SLIC to be promising, delivering a good balance of rendering performance, memory use (RAM), and storage space (flash). This work led to two ideas to make SLIC even better:

• Optimize palettes to make the 8-bit SLIC profile more efficient, especially for RGB565 destinations
• Support an alpha channel in the RGB565 profile

We have prototyped both of these features and prepared a document to explain the proposals in depth. The encoder changes are in the dev branch of this fork.

[phoddie]

Any interest or feedback?

[bitbank2]

Hi Peter, Sorry for the delay in evaluating your proposals. I agree with half of them :) I think the 8-bit palette mode should allow for variable length palettes, but not a RGB565 palette. I think that should be up to the app doing the decoding to convert the palette from RGB24 to RGB565 at the start fo the image decode. As far as the RGB565A format, I think it's unnecessary and will be detrimental to the compression of the pixel data. A simpler idea is to just create a new ALPHA8 profile that's the same as the current 8-bit profile, but with no palette. Since you can decode each stream incrementally, you can use the same ALPHA8 profile together with the 8-bit, 16-bit or 24-bit image data.

Thoughts?

[phoddie]

Hey @bitbank2, thanks for the response.

I agree with half of them :)

I think we're a bit under half, but appreciate the optimism. Let's see what we can do.

From your response, I infer that we have different use cases in mind. When you reached out to me about SLIC, you suggested it might be interesting for user interface elements in the Moddable SDK. I approached it from that perspective.

As you will understand from your own extensive work in embedded systems, RAM is often an extremely constrained resource. To be able to build rich user experiences on constrained devices like ESP8266, ESP32, and Raspberry Pi Pico, the Moddable SDK keeps as much data as possible in flash storage. For example, on an ESP8266 there is not enough memory to hold decompressed SLIC images of any useful size to build a user interface. Even on deices with more memory, storing uncompressed bitmap images quickly depletes RAM, putting pressure on other features such as networking. Our Poco rendering engine typically reads data from flash, decodes it, and composites it in a single pass. We apply this for a couple of other compressed data formats already, a run-length variant and a color cell variant. I approached SLIC with the same goal.

That approach makes it impractical to keep a separate bitstream for alpha or to store the palette in RGB24. You may not find our use case appropriate for SLIC, and I completely respect you wanting to keep a focused scope. Still, with the relatively minor modifications proposed SLIC works very well our approach in the Moddable SDK.

Let me make a couple of suggestions that might help us to find common ground.

1. Regarding the storage format of the palette, there is an option which would achieve your goal of keeping all 24 bits and my goal of having the 565 format available directly. Each entry can still be 3 bytes. The first two bytes would be the color in RGB565 format. The third byte is the remaining bits to increase the color resolution to RGB888 (so the low RGB323 bits). This would shift the responsibility to do additional work to the cases where a bigger pixel format is used, which presumably would often also be more powerful devices.
2. There's no question that my proposal reduces encoding efficiency. It was necessary to find an extra opcode, since the opcode space was already full. I made a guess that the "bad run" opcode would be the least impactful place to do this. Perhaps there is a threshold below which the impact is acceptable in exchange for the added functionality? When working on image compression standards (ISO MPEG-4 and JPEG-2000) I recall that it was common for the researchers to maintain a corpus of test images to evaluate the quality and efficiency of proposed encoding changes. I imagine you've done this to guide your development of SLIC. I would be interested to understand the overall impact of the change on target SLIC images.

[phoddie]

This proposal appears to be dormant. I want to confirm that it is due to a lack of interest and not because of a lack of time to review it. If the former, I'll close this proposal out.

[bitbank2]

Sorry for the delay; there are some bugs in SLIC encoding that someone found and I still need to find time to fix them. I like your RGB565/RGB323 palette idea. It might be more efficient to encode the used colors as RGB565 followed by the same number of bytes as RGB323 to keep the 565 case from needing to do anything special, but the 888 case would need to do a little bit of shifting around. Let's see if I can implement this some time this week.

[phoddie]

Thanks for the reply. I agree on the organization of the palette - full RGB565 bit palette followed by the delta bytes for RGB888. That's what I had in mind, but didn't express it well.

[phoddie]

Just checking in to see if this is going to move ahead or I should close this out as a rejected proposal. Please let me know. Thanks.