Feature request: 256-color image rendering for BlockImage

[danschwarz]

One feature I'd like to have is to (when necessary) render images using unicode half-block and ANSI 256-color mode rather than truecolor mode. This is for terminals (mainly Apple's Terminal app) that don't honor truecolor mode. 256 color rendering could be selected programatically by the calling application. I don't think there's any automatic or manual fallback to 256 color mode in the code now, but I can't test on Apple Terminal because my Mac isn't working right now :(

If it's helpful, you could require that all images to be rendered in 256 color mode should be preconverted to a palette-mode image using the XTerm 256 color palette before being passed to AutoImage. I've already got code to do that conversion.

[AnonymouX47]

Thanks.

I've actually had the necessary logic (with details of implementation) for this ready for a while in #61. Also, I intend to incorporate chafa.

Implementing the former actually isn't much work to be sincere but there is just one reason why I haven't done it.

A while back, I envisioned (now fully planned out) a new API that would open up the door to many new features and possibilities. Recent work (#88, #89) has been towards this. In fact, I had conceived this before the urwid support but I detoured since I almost didn't have it planned out at all then.

Implementing the feature requested here first would have two major effects:

1. Delay the implementation of the new API
2. Complicate the implementation of (or internal transition to) the new API

On the other hand, implementing the new API first will make the implementation of this feature slightly simpler because it will put in place certain required interfaces (e.g #62 amongst others).

Anyways, not to fear, this is included in the 0.7.0 milestone. So, it'll definitely happen soon. Current ETA is two weeks from today (no promises but I'll try my best to keep to it).

NOTE: I'm not insinuating that you want me to implement this first, I'm just given my reasons for not doing so.

[AnonymouX47]

Sorry, I should've done this earlier.

Please, I'll greatly appreciate a description or procedural breakdown of your proposed approach. Even though I already have a proposed approach, nothing stops me from modifying it or selecting your approach all together.

Code may work but I'll prefer something that can be understood by anyone without knowledge of a specific programming language.

Having this now will give enough time to ponder upon things and finalize the approach before it's time for implementation (as I explained earlier).

Thank you. 🙏🏾

[danschwarz]

Happy to explain. My intention is that users who launch the Toot application will have the ability to include a –256 option to force images to be rendered using 256 colors rather than truecolor, which would otherwise be the default. Apple Terminal users would want to include this option, as that terminal doesn't support truecolor.

From the app coder's point of view, I want to be able to pass any Pillow image to the widget, of any bit depth, and have it rendered at my chosen bit depth. When rendering with 8 bit depth, the widget should convert images to the standard Xterm 256 color palette before rendering. (You might want to allow custom palettes as an option, but I'm going to stick with the standard.)

For completeness you may want to support 88 color mode, 16 color mode, and monochrome as well. But I'm not sure I'll support those in the Toot app; I may just include an option to suppress image display entirely for users with monochrome or 16 color displays.

That's about it. One other thing I've considered is auto detection of truecolor and 256 color support, but this is probably not going to be reliable enough (false negative/no result on terminals that don't support reading the background color).

Let me know if I can provide any additional detail. Thanks!

[AnonymouX47]

Thanks.

First, i should note that everything you've said is useful. Anyways, what I asked for was your approach to implementing 256-color support (since you mentioned already having some code).

---

(You might want to allow custom palettes as an option, but I'm going to stick with the standard.)

Hmm... I'll need to look a little more into this. One thing I know for sure is for this to give expected results, it'll require modifying the terminal pallete to match the pallete used for the quantization (mapping the 8-bit colors to an index).

Included in my plans is a set_terminal_pallete() method to set the terminal pallete using OSC 4 but I only had intentions for a single pre-defined pallete, no custom ones. I'll look into extending that then.

For completeness you may want to support 88 color mode, 16 color mode, and monochrome as well. But I'm not sure I'll support those in the Toot app; I may just include an option to suppress image display entirely for users with monochrome or 16 color displays.

I intend to add support for, 3-bit (8) color and No color, i guess i'll 16 add 88 to the list.

That's about it. One other thing I've considered is auto detection of truecolor and 256 color support, but this is probably not going to be reliable enough (false negative/no result on terminals that don't support reading the background color).

Yeah, this. That's actually what #62 is about (though it doesn't actually define support detection for any mode, it only defines the interface required to implement support detection).

Anyways, I have some thoughts concerning truecolor (already implemented actually) and 256-color support detection... but like you've said, these methods are not reliable (but that's all we got).

---

Looking at all these, it seems color quantization (i.e mapping to a palette) is what I'll end up going for.

My original "calculation" approach is not flexible (can't allow for custom palettes) and works only if the colors in the terminal's palette are evenly distributed (yet to investigate how true or not true that is but I've come across at least one source indicating terminal 6-by-6 color cubes (232 colors) are not evenly distributed).

I'll be expecting more on what i mentioned in my second paragraph.

Thanks.

[danschwarz]

Re: bit depths, if you are using chafa then go by what it supports.

Re: palettes, It’s friendlier to use the current palette than to change it behind the application’s back. Defaulting to the xterm palette seems safest. But if you can read the current palette then quantize the image into that palette, that’s probably ok too? Not sure how many terminals implement the pallet reading extension. Chafa may have it’s own way of dealing with palettes, ¯_(ツ)_/¯

On Sat, Jun 3, 2023 at 3:24 PM AnonymouX47 @.***> wrote:

Thanks.

First, i should note that everything you've said is useful. Anyways, what I asked for was your approach to implementing 256-color support (since you mentioned already having some code).

(You might want to allow custom palettes as an option, but I'm going to stick with the standard.)

Hmm... I'll need to look a little more into this. One thing I know for sure is for this to give expected results, it'll require modifying the terminal pallete to match the pallete used for the quantization (mapping the 8-bit colors to an index).

Included in my plans is a set_terminal_pallete() method to set the terminal pallete using OSC 4 but I only had intentions for a single pre-defined pallete, no custom ones. I'll look into extending that then.

For completeness you may want to support 88 color mode, 16 color mode, and monochrome as well. But I'm not sure I'll support those in the Toot app; I may just include an option to suppress image display entirely for users with monochrome or 16 color displays.

I intend to add support for, 3-bit (8) color and No color, i guess i'll 16 add 88 to the list.

That's about it. One other thing I've considered is auto detection of truecolor and 256 color support, but this is probably not going to be reliable enough (false negative/no result on terminals that don't support reading the background color).

Yeah, this. That's actually what #62 https://github.com/AnonymouX47/term-image/issues/62 is about (though it doesn't actually define support detection for any mode, it only defines the interface required to implement support detection).

Anyways, I have some thoughts concerning truecolor (already implemented actually) and 256-color support detection... but like you've said, these methods are not reliable (but that's all we got).

Looking at all these, it seems color quantization (i.e mapping to a palette) is what I'll end up going for.

My original "calculation" approach is not flexible (can't allow for custom palettes) and works only if the colors in the terminal's palette are evenly distributed (yet to investigate of true or not true that is but I've come across at least one source indication terminal 6-by-6 color cubes (232 colors) are not evenly distributed).

I'll be expecting more on what i mentioned in my second paragraph.

Thanks.

— Reply to this email directly, view it on GitHub https://github.com/AnonymouX47/term-image/issues/90#issuecomment-1575144604, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY4FJRZHMM4SMG4AMVKQT3XJOFNTANCNFSM6AAAAAAYTFQZ2M . You are receiving this because you authored the thread.Message ID: @.***>

[danschwarz]

Oh, my approach to implementing 256 color support was as follows:

Fetch image via https If 256 color option is set, quantize image to 256 color palletized image using Pillow functions and Xterm standard palette

Pass image to ANSIWidget (my block rendering widget)

the widget checks bit depth of incoming Pillow image.

• if truecolor, render using truecolor
• If palettized, render using 256 color

I didn’t build support for 1 bit or other depths because these weren’t needed in my use case.

[AnonymouX47]

Re: bit depths, if you are using chafa then go by what it supports.

Yeah, I sure will... but the thing is, i also intend to add support for other rendering styles Chafa doesn't currently implement. Happily, Chafa does support all the color modes mentioned above (I think with even a couple more).

Re: palettes, It’s friendlier to use the current palette than to change it behind the application’s back. Defaulting to the xterm palette seems safest. But if you can read the current palette then quantize the image into that palette, that’s probably ok too? Not sure how many terminals implement the pallet reading extension. Chafa may have it’s own way of dealing with palettes, ¯_(ツ)_/¯

What do you think about this?

• Try to read the terminal the terminal's palette (that's already partially implemented actually).
  • If supported, use that as the default.
  • Otherwise, use the pre-defined XTerm palette as default.
• Allow the use to set a custom pallete, if setting the pallete is supported by the terminal emulator.
  • Will never modify the palette behind the application's back. Will be on request.

I'll look into how Chafa goes about palettes (or probably just ask).

[AnonymouX47]

Oh, my approach to implementing 256 color support was as follows: Fetch image via https If 256 color option is set, quantize image to 256 color palletized image using Pillow functions and Xterm standard palette Pass image to ANSIWidget (my block rendering widget) the widget checks bit depth of incoming Pillow image. - if truecolor, render using truecolor - If palettized, render using 256 color I didn’t build support for 1 bit or other depths because these weren’t needed in my use case.

Hmm 🤔 ... I think a slight variation (and extension to other color modes) of this will do the job.

Thanks

[AnonymouX47]

Please, don't mind the change the change in version milestone... the schedule remains the same.

I just realized changes would be difficult to keep track of with what next couple changes... the CHANGELOG would become really messy if I decided to release all at once.

Thanks for your patience 🥲 🙏🏾

[AnonymouX47]

Hello @danschwarz!

It's definitely been a while (Just scrolled up now to check and realized it's just been a year. Oh, how time flies!).

Of recent, you reminded me again of this issue and I decided to take a night (plus a couple more hours) off to do the needful in #109 (please, take note of the admonition there). I hope this would suffice until #61 is resolved. This is what it looks like in a couple TEs by running the following snippet:

from term_image.image import BlockImage
BlockImage.set_render_method("INDEXED")  # See the docs for other ways to achieve this.
image = BlockImage.from_file("images/python.png")
print(image)

XTerm	WezTerm	Konsole

Looks like this in urwid (with the UrwidImage widget):

You can install from the indexed-color branch using:

pip install git+https://github.com/AnonymouX47/term-image.git@indexed-color

I'd appreciate it if you can test out the feature (especially on a terminal emulator with no truecolor support) and give some feedback. I'd be even more glad if you can test it on the default Mac OS terminal in particular, as I don't have immediate access to a Mac. For this, you might wanna use this version of docs (especially, the BlockImage docs) since the changes aren't merged yet.

As soon as you confirm everything is working as expected/desired, I'll go ahead to merge and release since there's nothing else I have in mind for v0.7.2. If you have any inquiries or any other suggestions, bring 'em all in please.

Thank you so much for your suggestions and most importantly, your patience. :pray:

[danschwarz]

Thanks for implementing this! I'll try to test it over the weekend, including on MacOS.

[AnonymouX47]

You're welcome. :smiley:

I'll be expecting your feedback. Thanks.

[danschwarz]

So I'm testing it with an updated version of the toot code, the only change is that I'm forcing BlockImage.set_render_method("INDEXED"). Here you can see two overlapping windows. The frontmost window is MacOS Terminal, and the back window is iTerm on Mac.

iTerm2 supports truecolor mode. The image you see in the iTerm window is 256-color dithered, so that's good. Interestingly, the window title of the iTerm window has a part of a terminal command sequence in it, not sure what that's about.

But more importantly, the Terminal window is showing garbage as before. Based on this behavior, my guess is that it is sending truecolor control sequences to the terminal, even though it's dithering down to colors from the 256 color palette. I think that's why it's working in iTerm2 and not Terminal. Per your docs, it should be using the indexed color control sequences.

[AnonymouX47]

Thanks so much for taking the time to test and for the feedback. :smiley:

So I'm testing it with an updated version of the toot code, the only change is that I'm forcing BlockImage.set_render_method("INDEXED").

If you don't mind, please tag me on the PR when you open it. I might have one or two suggestions.

Interestingly, the window title of the iTerm window has a part of a terminal command sequence in it, not sure what that's about.

Yeah, turns out that's because support was being forced for all image render styles here. I've fixed that in https://github.com/ihabunek/toot/pull/484/commits/1ea2e29e2563f34cedfbe1f15ad7260684f3b84e.

Note: If there are any conflicts between my PR and yours implementing 256-color support, I'm willing to resolve them.

But more importantly, the Terminal window is showing garbage as before. Based on this behavior, my guess is that it is sending truecolor control sequences to the terminal, even though it's dithering down to colors from the 256 color palette. I think that's why it's working in iTerm2 and not Terminal. Per your docs, it should be using the indexed color control sequences.

Turns out it's most likely due to my use of the standard (colon-delimited) indexed-color control sequences, just like in https://github.com/AnonymouX47/term-image/commit/b38e2e84d9d9f9102800161860683764e730bd28#commitcomment-115569596. Anyways, I've now switched to using the legacy (semi-colon-delimited) sequences.

You can please test with the updated indexed-color branch and let me know the results.

Thanks once again. :pray:

[danschwarz]

Testing with the updated indexed-color branch was successful. Here you can see an indexed color image in Terminal, and the same image in iTerm. (It looks substantially worse in Terminal due to the lousy default font used by Terminal, but that can be changed by the user.). This is using toot code without your recent PR, which accounts for the command sequence in the upper left corner of the Terminal screen.

[AnonymouX47]

Glad to hear that!

I'll make a release as soon as possible.

Thanks. :smiley:

[AnonymouX47]

I have just released v0.7.2 with this feature.

Thank you very much. :smiley: