Direct drawing API for SIXEL and such

[diamondburned]

Recently, many new, modern terminals are popping up with decent SIXEL support, and with VTE's SIXEL branch coming closer to being ready, I've been thinking of various ways of adding SIXEL graphics through tcell to ultimately make an image widget in tview; however, I'm making an issue first to see how such a thing should be best implemented.

My current idea involves these interfaces:

// DrawInterceptFunc is the callback to intercept a draw. It is called when the
// lock is acquired. Sync is true if the whole screen is about to be cleared, or
// for after interceptors, after clearing.
type DrawInterceptFunc func(sync bool)

// DrawInterceptAdder is an interface that Screen can implement to add a draw
// itnerceptor function. This interface could be used to draw custom SIXEL
// images as well as other raw terminal things.
//
// Implementations must call these functions before any state changes, or in teh
// case of After, after all state changes. For example, draw interceptors should
// be called before the cursor is hidden, and the after interceptors should be
// called after the cursor is shown and everything is drawn into the buffer (but
// before flushing).
type DrawInterceptAdder interface {
    // AddDrawIntercept wraps the existing draw intercept function with the
    // given one.
    AddDrawIntercept(fn DrawInterceptFunc)
    // AddDrawInterceptAfter adds the draw interceptor after everything is
    // drawn.
    AddDrawInterceptAfter(fn DrawInterceptFunc)
}

// DirectDrawer exposes the raw draw function for the caller to draw code
// unknown to tcell.
type DirectDrawer interface {
    // DrawDirectly draws the given bytes as-is directly into the screen.
    DrawDirectly([]byte)
}

They have a problem, however. Callbacks may call methods on Screen, which would cause a deadlock, because the lock would already be acquired in draw() calls, which is where these callbacks are executed. I'm currently thinking of using a halting mutex (mutex that allows invalidation of its methods temporarily while it's acquired), but this seems hacky.

The idea of unlocking and relocking may seem fine at first, but it will lead to excessive graphical blinking.

The ultimate use case of this type of API would be drawing SIXEL graphics onto tcell applications, like so:

For the record, this image is rendered on the foot terminal using my current prototype branch.

[diamondburned]

With the current changes in my branch, I was able to make a working SIXEL implementation with GIF support.

[ferdinandyb]

Hi! Is there any plans on moving forward with integrating this? I've started looking into adding image viewing to aerc which uses tcell-term and according to the author, the easiest would be if tcell would support it directly, because otherwise it will be a bit hacky.

[gdamore]

I haven't looked at this in quite a while. I'll need some time to re-review it.

My biggest complaints right now with these relate to a couple of areas of concern:

• for SIXEL specifically, making sure that it fits in our model sensibly -- bitmap graphics is way different than character cells.
• identifying which terminals support this -- the problem with that is that there are a bunch of different variants, and even worse, different versions of those variants, and so using $TERM to identify this capability becomes increasingly challenging. This is fine when adding a feature is mostly benign if the terminal doesn't support it, but this seems unlikely to be the case for SIXEL.

Anyway, I'll look again, maybe with a fresh perspective.

[gdamore]

I've reviewed the PR for this, and had some thoughts. It needs work before it can land, and some of it needs changes in approach, because SIXEL doesn't really work for all terminals -- it is linked intrinsically to the model of a stream of bytes and escape sequences, and that isn't how some of the targets we support work (neither Windows console nor the new HTML terminal widget).

I think that there are however ways forward to achieve what is desired here, in a way that doesn't lead to unacceptable compromises. My notes on that are in the PR.

[gdamore]

Note that the approach discussed here doesn't actually offer full SIXEL support, but merely exposes some things that make it possible for a SIXEL library to coexist with tcell.

[ferdinandyb]

Thanks for looking into this!

[TeddyDD]

Is it possible to draw sixel image with transparency?

[gdamore]

I actually think the PR to enable this was merged already. I don't know whether transparency is possible here but I sort of doubt it.

[diamondburned]

Closing since https://github.com/gdamore/tcell/pull/602 is merged.