Support 1337 escape sequence to show inline images

[cristynkells]

Details

• Browser and browser version: all
• OS version: all
• xterm.js version: 2.4.0

Feature request

Is there a way to view images from the command line?

by either clicking on the image (much like the linkify addon) or via a command:

$ cat vacation-beach-pic.png

[Tyriar]

Clicking on the link to open the image is possible via the an API that's experimental at the moment. It's experimental because it's likely to change but here is the current syntax:

// Add a link for a particular regex and handle when it's clicked.
xterm.registerLinkMatcher(/\w\.png/, function (mouseEvent, imageFile) => {
  // imageFile is the text of the link
});

The API should hopefully be finalized in the next couple of versions.

[cristynkells]

Hi Daniel, thank you for your speedy reply! Was wondering if there a way to display html on the fly through a set of control characters? Similar to how DomTerm uses “\e]72;\a” sequences ?

https://lwn.net/Articles/670078/

From: Daniel Imms notifications@github.com Reply-To: "sourcelair/xterm.js" reply@reply.github.com Date: Saturday, March 18, 2017 at 4:47 PM To: "sourcelair/xterm.js" xterm.js@noreply.github.com Cc: cristynkells cristynkells@gmail.com, Author author@noreply.github.com Subject: Re: [sourcelair/xterm.js] how to view images in line (#614)

Clicking on the link to open the image is possible via the an API that's experimental at the moment. It's experimental because it's likely to change but here is the current syntax:

// Add a link for a particular regex and handle when it's clicked. xterm.registerLinkMatcher(/\w.png/, function (mouseEvent, imageFile) => {   // imageFile is the text of the link }); — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Tyriar]

@cristynkells there isn't right now, standard xterm doesn't support anything like that so if we did want to support it it would certainly need to be in an addon.

[DonJayamanne]

@Tyriar

it would certainly need to be in an addon

Is this supported today? Right now I don't see how this can be done even with an add on. Basically I'm trying to investigate what it would take to build something with inline plots (something we already have in other IDE, PTVS, RTVS, Spyder, and the like e.g. https://ipython.org/ipython-doc/3/interactive/qtconsole.html).

[Tyriar]

@DonJayamanne since this issue I've become aware of a standard some terminals have implemented for displaying images. Because of this we could add it to the core library, it's probably a pretty hefty feature though in terms of making sure everything is correct, particularly relating to getting the scroll bar right as right now there's no such thing as variable height rows in the viewport.

[DonJayamanne]

implemented for displaying terminals

Yes, I found iTerm does this.

Thanks for getting back.

[ebraminio]

Hi there. I opened a related issue on VSCode and redirected here. I wonder if supporting iTerm2 image feature is a possibility as it can be used as easy as this,

printf "\n\033]1337;File=;inline=1:`cat a.png | base64`\a\n"`

And I already used it on a project.

[ztmr]

@ebraminio If we are going to draw graphics and possibly some more hypertext, I would really like to be able to specify coordinates where to render it, not just blindly dump it under the current cursor position... check #1176.

[rs]

@ztmr: You can use other termcap commands to move the cursor and display the image wherever you want. That is what I do in jplot to redraw the image in place.

[ztmr]

@rs: Oh, of course, how could I forget about CUP! :o)

[Tyriar]

Opps this was labelled incorrectly. This would be a great feature to get a contribution for, here are some of my thoughts:

• Do we need some pluggable way to support escape sequences?
• Maybe this should live inside an addon?
• Currently every single row is a fixed length, are images to be shown fixed to this grid as well or should the size be different? For example with row height=10px and image height=15px, should the image consume 15px or 20px on the terminal). If the latter we may want to tackle https://github.com/xtermjs/xterm.js/issues/1140 first.

[jerch]

+1 for custom escape sequences. Best place to start with those is to implement OSC and DCS subparser interfaces, where custom subparser can hook into the main parser. Also the main parser still has some escape code flaws and the test library still needs to be completed to cover all the basic sequences.

Btw DEC already had some terminal extensions for graphics and image loading (REGIS and SIXEL). No clue if it would make much sense to implement those, since the browser env of xterm.js has much better capabilities to draw such things.

[jerch]

SIXEL is supported by some programs (gnuplot) and some terminals (mlterm, xterm), there is even libsixel (https://github.com/saitoha/libsixel) for easy conversion. Seems not to be as dead as I thought...

Unsure about the OSC 1337 command, this seems to be supported by iterm2 only.

Edit: Also see jupyter console and https://github.com/liftoff/GateOne, they are both capable to show graphics, not sure though if they hack it around/on top of the terminal on a higher level. This might be a nice addon feature for xterm.js, still in doubt if this should be part of the terminal itself, as it raises many questions from data and state to output representation.

[soccermitchy]

From my point of view, Sixel support would be nice for a monitoring dashboard-style thing. Could easily show gauges and stuff in it - there are even browsers, iirc, that support rendering with it.

[Tyriar]

Here are some screenshot of how image placement/selection works in iterm2, not totally sure how the margins work yet.

[jerch]

I think SIXEL is the way to go for images, also made some progress with an early SIXEL parser (https://github.com/jerch/node-sixel). Once we are settled with the new buffer layout and marker/decorator rework this should be straight forward to integrate.

[PerBothner]

While sixel is a "standard" (and I think it is desirable to support it), it is a rather obsolete standard. It is not well suited to today's hardware, software, or workflow. Very few people have heard of it, and very few programs produce or consume it. It is extremely inefficient for photographic images (does not support many colors well), and it is constrained to images whose height is a multiple of 6.

The concept (if not necessarily the specifics) of the 1337 escape sequence is a lot easier for most people (and programs) to work with: a base64-encoding of standard current image-file formats: png, gif, jpeg, and optionally svg. I think that is the way to go. The 1337 syntax is a bit clunky (the inline option defaults to 0 - huh?), but it works. Is there some other escape sequence with similar functionality used by other terminals?

(While DomTerm doesn't support the 1337 escape sequence, it supports the same idea, except you have to wrap the data in an <img> or <svg> html element. Adding support for the 1337 escape sequence should be trivial.)

[jerch]

@PerBothner Imho the ticky part is #1901, as soon as we solved the question how to deal with the grid and layering for "foreign boxes" I see no reason why we should not support image formats that are natively supported by the browser env. But for that I'd love to see a broader discussion (on @egmontkob's platform?) and maybe rethink the sequence. For sixel the sequence is already there for 30ys, still the correct handling of the representation state is wonky (again #1901).

[PerBothner]

I fully agree with @jerch's most recent comment.

[PerBothner]

I created an issue on the freedesktop.org terminal-wg to try to standardize "simple image display". I hope we can get agreement on a suitable semi-portable escape sequence. Possibly the 1337 escape sequence (though it has some issues); possibly something else.

[sdegutis]

I think this can be broken down into two issues:

1. Handling of arbitrary escape code sequences, either internally or externally (via API). This is a useful feature for more than just images, and is orthogonal to the concept of what is shown or even how it's shown. It just needs a function callback API.

2. Hosting of foreign DOM content, in this case an image, but other content has the same properties: a DOM element with an x,y,w,h relative to a sub-grid of cells on the terminal's character grid.

Both of them would help me a lot, but in different ways. It might be worth creating two new issues for each.

[sdegutis]

• Currently every single row is a fixed length, are images to be shown fixed to this grid as well or should the size be different? For example with row height=10px and image height=15px, should the image consume 15px or 20px on the terminal). If the latter we may want to tackle #1140 first.

I just assumed the image would still be 15px and the extra 5px would be empty, so that the image would be "overlayed" over the rows. Considering cell width/height is so small, I don't see any drawbacks to the slight wasted pixel space of this solution.

[Tyriar]

Handling of arbitrary escape code sequences, either internally or externally (via API). This is a useful feature for more than just images, and is orthogonal to the concept of what is shown or even how it's shown. It just needs a function callback API.

Already done:

https://github.com/xtermjs/xterm.js/blob/bd0d267d972a1aab4777cddecf9bf8196d7aa44f/typings/xterm.d.ts#L1060-L1127

My comment is probably a little out of date, @jerch has been the main one looking at images with this, https://github.com/xtermjs/xterm.js/pull/2503 and https://gitlab.freedesktop.org/terminal-wg/specifications/issues/12

[sdegutis]

Ah right, yeah I even use that API. Whoops, should have remembered. Thanks.

[tracker1]

I do think it would be cool to support sixel as well as 1337 sequence... I do think that whatever is displayed should be potentially resized if too wide, and the space taken padded to a full line height.

May already be possible with the extension, not sure if custom handlers can hook into a given set of columns/rows and the height-width of either to custom render several rows?

Came into this after discovering sixel formats and noticing that it isn't available in the terminals I use most (one of which is vs code).

[jerch]

@tracker1 See #2503 for the current development state.

[trbot86]

I really hope this is still in active development. (Seems like a PR is stalled waiting for merge into xterm master?)

As a scientist who writes code to run experiments that produce graphs, the ability to visualize these graphs (in VSCode's xterm) without opening external software would be huge.

[jerch]

I really hope this is still in active development. (Seems like a PR is stalled waiting for merge into xterm master?)

Well, originally I intended to add the SIXEL part as sync code (which is almost feature complete), but the image decoding puts quite some pressure on the main thread, which we cannot simply ignore in bigger embedding apps like vscode as it would lead to bad lag spikes. The addon is basically waiting for #3222 to get finished, with that PR we can offload the hard work to a worker thread.

As a scientist who writes code to run experiments that produce graphs, the ability to visualize these graphs (in VSCode's xterm) without opening external software would be huge.

Yes scientific REPLs with direct graphics dumping caps will benefit alot from this.

[trbot86]

That makes perfect sense, and I'm really glad to hear it. Looks like you're down to just MacOS specific test failures, but if the skillset of a CS concurrency/multicore researcher would be useful, feel free to let me know.

[jerch]

Well #3222 will take abit more time, it changes several crucial aspects of the parser handlers which needs proper testing. Once it is there the adoption needs in #2503 should be small.

if the skillset of a CS concurrency/multicore researcher would be useful, feel free to let me know.

For #2503 I wonder, how much functionality could/should go to the workers in the end. I really would like to use OffscreenCanvas, but thats still unlikely to happen, as the API is far from stable across browser engines. If you have more insights/experience regarding this, fee free to discuss the internal addon shaping over there.

[CoconutMacaroon]

Any update on this? Having support for Sixel would be helpful for me.

[jerch]

Sixel support can now be tested with the xterm-addon-image npm package (see https://github.com/jerch/xterm-addon-image for some docs, how to use it).

[trbot86]

So now that we have sixel support in xterm.js, is there any chance we can use this version of xterm.js for the vscode integrated terminal? :) (As I recall, @Tyriar said that conversation should continue here.)

[Tyriar]

@trbot86 I think we could start experimenting with that yes, it's just a matter of finding the time. I'll bring it up in my planning discussion this week.

@jerch btw I created a png decoder recently which is fast and very well tested which you might prefer over png-ts 😉 https://www.npmjs.com/package/@lunapaint/png-codec, I consume it as a git submodule so it's possible you might hit packaging problems that we'd need to work out.

[jerch]

@jerch btw I created a png decoder recently which is fast and very well tested which you might prefer over png-ts wink https://www.npmjs.com/package/@lunapaint/png-codec, I consume it as a git submodule so it's possible you might hit packaging problems that we'd need to work out.

Thats great news. Well, I am actually looking for a fast PNG decoder to be used for the next protocol extension of the module to directly support png and jpg formats (tbh sixel is a really poor format choice). Would your decoder work in a worker context, or does it need browser mainthread-only APIs? And do you happen to have a fast JPG decoder as well by any chance?

[Tyriar]

@jerch I think it will work in a worker, if it doesn't I'd want it to. I currently use it in node and main thread browser without problems.

And do you happen to have a fast JPG decoder as well by any chance?

I will be making one of these eventually, decode to help understand the format and then encode to give full control over how it's written. Currently doing TGA since basic read/write of jpeg is possible via canvas 🙂

[jerch]

@Tyriar Will def. check it out for the next image formats in the addon, which then also should provide proper serialize/deserialize functionality (currently not possible with sixel only). In general the GIP protocol discussion is somewhat stalled, so I might come up with an interim solution addressing most issues we already agreed on. For jpg decoding in a worker I think I cannot rely on canvas yet, as it is not officially supported by safari? Well, its like waiting for ~Godot~ Safari to close certain implementation gaps... :sweat:

[dvag-sebastian-goeb]

Any progress on this? https://github.com/xtermjs/xterm.js/pull/2503 seems to have been parked. I would really love to see inline images in VSCode especially (for Jest + Puppeteer + Jest-Image-Snapshot screenshot testing in my case).

[markusbkk]

Bump

I'd be interested in this, as well.

[jerch]

@dvag-sebastian-goeb @markusbkk There is inline graphics support with xterm-addon-image addon (sixel only atm).

[markusbkk]

@dvag-sebastian-goeb @markusbkk There is inline graphics support with xterm-addon-image addon (sixel only atm).

I saw.

Unfortunately, that's not quite the same as iTerm2. I'm working on something that requires iTerm2 image protocol support.

[blacktop]

bump 🙏

[jerch]

@markusbkk PR for iterm IIP support: https://github.com/jerch/xterm-addon-image/pull/43.

[jerch]

Just merged the IIP support. It passes all my basic tests, but since I have no mac to work with I cannot really compare it with iTerm and therefore labelled it as alpha. So if anyone wants to do some digging - a side by side comparison with iTerm's output from different images and imgcat settings would be really helpful.

[wsshin]

I landed here from the VS Code issue https://github.com/microsoft/vscode/issues/65283. Could you describe how to test xterm-addon-image in VS Code? I am willing to test it on my Mac.

[jerch]

@wsshin Idk if vscode allows loading custom xterm.js addons - if not, you prolly have to pull source and add it there yourself.

[Tyriar]

@wsshin if you want to try test it out in vscode you can search the vscode codebase for xterm-addon-search which is an example usage of an addon from npm

[trbot86]

thanks for the tip @Tyriar and the addon @jerch. it was my first time even compiling vscode, but i managed to hamfist xterm-addon-image into vscode: https://github.com/microsoft/vscode/compare/main...trbot86:vscode:xterm-addon-image

it seemed like i had to start with an old vscode commit (59faab44cd8287e247cfab8c389b9f0f44174bcf) to satisfy the xterm version dependency in xterm-addon-image. there's probably a better way to do this. i'm just a newbie to the code base, node.js, and typescript.

but man it's awesome to see it work!

[jerch]

it seemed like i had to start with an old vscode commit

This is prolly due to my rigid versioning policy in the addon:

  "peerDependencies": {
    "xterm": "~5.1.0"
  }

which basically tags addon releases to a certain xterm.js minor version (normally most recent one). This close addon entanglement to xterm.js versions is needed due to its heavy usage of private APIs (which also cannot be exposed by public API easily). This is somewhat unfortunate, but I dont know of a better way atm. If you are sure it will work with current master (most of the time the last addon release can be carried forward until a crucial change in xterm.js happens), then you can raise the version to the next minor number of xterm.js (imho beta releases point to the next minor already, and vscode uses those).

There is a major catch though with vscode - in the past vscode used the writeSync method of xterm.js for certain tasks. If thats still the case, then you will have to disable IIP support. Background: The IIP protocol path uses an async sequence handler, which will create havoc on the terminal state, when used along with the sync write method. Since this is the case for all async handlers, writeSync got deprecated with a big warning several versions ago.

[trbot86]

thanks for that remark. i haven't previously considered disregarding or updating the version requirement of xterm-addon-image. it turns out vscode:HEAD (specifically 10b2a8e42eddf7504f7b6e6b19d73eb5f888d075) compiles and works with xterm-addon-image, even though it's running xterm 5.2.x. i've updated my repo accordingly.