Support 1337 escape sequence to show inline images

[cristynkells]

Details

• Browser and browser version: all
• OS version: all
• xterm.js version: 2.4.0

Feature request

Is there a way to view images from the command line?

by either clicking on the image (much like the linkify addon) or via a command:

$ cat vacation-beach-pic.png

[trbot86]

is there any chance of this making it into insiders? i hadn't previously used vscode OSS, and it's starting to set in how limiting it is... not having access to extensions renders it useless...

edit: i realize you can sideload extensions, but you can't evidently can't make remote-ssh work, and i do 100% of my work through the remote-ssh extension. archlinux guys have patches for this, but i'm trying to make windows OSS work, and there doesn't seem to be any workaround that i can see for the missing remote-ssh functionality.

edit2: if anyone wants to try to reproduce or improve what i've done, the head commit is whatever junk i've been messing around with, and might be broken. the "good" commit is https://github.com/trbot86/vscode/commit/5799f440141cf7182f4c20e6e6b6743c4c0a8577

[jerch]

@trbot86 Plz note that there are still some loose ends with the addon - most prominent the ability to serialize image data forth and back. So I am not sure, if it is ready for production rollout yet. I am currently dealing with the serialization issue in https://github.com/jerch/xterm-addon-image/pull/48. This is still in conceptual phase, as there is no prior art to learn from, so it will take a few more weeks to get things sorted out.

[jerch]

@Tyriar Should we close this issue, as the addon is in the base repo now?

[trbot86]

I can't find a better issue in vscode (that isn't locked) to ask this in, but now that this is implemented in vscode OSS:main, how do we get this into insiders edition?

[jerch]

@trbot86 Isnt the insider edition built regularily from vscode's base repo? Because vscode's repo contains all the needed bits, so it should just work imho. Or do you mean something else with "insiders edition"?

[Tyriar]

@trbot86 images are now enabled by default in main, stable and insiders. See https://code.visualstudio.com/docs/terminal/advanced#_image-support

[trbot86]

Thanks for the link. I had tested the feature without success, but the image I was testing was extremely large (~8k resolution), and it produced no output. It works when using a smaller image.

Do you happen to know offhand how vscode's sixel support is intended to handle images that are too large to fit in the terminal window? Trying to isolate whether the failure is in the program converting the image to sixels, or in vscode or xterm.js.

Edit: I also want to say thank you!! This is amazing! Should have said that from the beginning. :)

[jerch]

... sixel support is intended to handle images that are too large to fit in the terminal window?

@trbot86 Not sure if you mean in terms of memory usage and/or screen coverage:

memory Since images grow rather big with higher resolutions and JS can only address up to 2GB - there a few limits in place to avoid OOM from exhaustive image usage by the addon:

• pixelLimit - default is 16M (e.g. 4096x4096 squared), a sequence trying to allocate more pixels gets cancelled
• sixelSizeLimit - default is 25MB, limits the max. size of a single sixel sequence, bigger sequences are again cancelled
• iipSizeLimit - default is 20MB, same as sixelSizeLimit but for IIP sequences

screen coverage Sixels are currently capped at 16K pixels in x-direction (truncated), but not actively capped in y-direction, only indirectly by pixelLimit. Thus it is possible to create an image with 16K x 1024 pixels up to an image with 1 x 16M (better dont try that, the browser engine prolly wont like such a canvas). Sixel pixels are mapped 1:1 to CSS pixels, image data beyond the viewport to the right is preserved (can be "un-covered" by resizing), excess to the bottom will normally scroll the terminal window. Sixels dont support a resizing policy as IIP does. Images from IIP have no additional hard caps, only pixelLimit. Furthermore IIP supports a resizing policy during image loading (e.g. width=50% to resize an image to 50% of the current viewport width).

Also see https://github.com/xtermjs/xterm.js/blob/master/addons/xterm-addon-image/README.md.

[trbot86]

Thanks for the detailed answer. It sounds like I'm likely butting up against pixelLimit, as the image in question was something like 8000x5000 pixels. I'm not sure how much the sixel format inflates the PNG size, but the image was something like 2-3MB. I think sixels are perhaps quite wasteful, so I could conceivably also hit the sixelSizeLimit. Well, I can probably just avoid trying to show such large images. It would be nice, from a graceful degradation standpoint, to perhaps render a small image explaining the failure when a limit is hit, rather than the actual image. No big deal though. :)

[jerch]

8000x5000 pixels? Thats 160 MB in RGBA for just one image - no, you cannot expect this to work in an env with only 2GB address space. If you have hi-res output needs a terminal is prolly not the right tool for it.

If you want reasonable output & performance, then it is better to grab the terminals viewport size (e.g. by CSI 14 t) and zoom & tile into your big image on app side just sending the area of interest. Also sixel is not a good choice for that, as it is very costly to encode (needs quantization & dithering) and has poor output quality - IIP is def. the better choice here (better quality with bigger color space, better compression, much faster to generate).

It would be nice, from a graceful degradation standpoint, to perhaps render a small image explaining the failure when a limit is hit...

Thats not possible while maintaining the text grid - it would screw up output / screen state. Also other terminals have similar limits in place, so it is not surprising behavior.

[trbot86]

Yes that's totally fair. It makes sense to slice or shrink large images before viewing them. Easy enough to add an imagemagick convert -resize step in a script.