Image output in xterm.js.
Version 0.4.x will be the last version from this single repo. Future versions will reside as addon in the xterm.js main repo.
npm install --save xterm-addon-image
The addon integrates tightly with the xterm.js base repo, esp. for tests and the demo.
To properly set up all needed resources see bootstrap.sh
or run it directly with
curl -s https://raw.githubusercontent.com/jerch/xterm-addon-image/master/bootstrap.sh | XTERMJS=5.2.0 IMAGEADDON=master bash
The addon sources and npm package definition reside under addons/xterm-addon-image
.
import { Terminal } from 'xterm';
import { ImageAddon, IImageAddonOptions } from 'xterm-addon-image';
// customize as needed (showing addon defaults)
const customSettings: IImageAddonOptions = {
enableSizeReports: true, // whether to enable CSI t reports (see below)
pixelLimit: 16777216, // max. pixel size of a single image
sixelSupport: true, // enable sixel support
sixelScrolling: true, // whether to scroll on image output
sixelPaletteLimit: 256, // initial sixel palette size
sixelSizeLimit: 25000000, // size limit of a single sixel sequence
storageLimit: 128, // FIFO storage limit in MB
showPlaceholder: true, // whether to show a placeholder for evicted images
iipSupport: true, // enable iTerm IIP support
iipSizeLimit: 20000000 // size limit of a single IIP sequence
}
// initialization
const terminal = new Terminal();
const imageAddon = new ImageAddon(customSettings);
terminal.loadAddon(imageAddon);
IMPORTANT: The worker approach as done in previous releases got removed. The addon contructor no longer expects a worker path as first argument.
By default the addon will activate these windowOptions
reports on the terminal:
to help applications getting useful terminal metrics for their image preparations. Set enableSizeReports
in the constructor options to false
, if you dont want the addon to alter these terminal settings. This is especially useful, if you have very strict security needs not allowing any terminal reports, or deal with windowOptions
by other means.
SIXEL Support
Set by default, change it with {sixelSupport: true}
.
Scrolling On | Off
By default scrolling is on, thus an image will advance the cursor at the bottom if needed.
(see cursor positioning).
If scrolling is off, the image gets painted from the top left of the current viewport and might be truncated if the image exceeds the viewport size. The cursor position does not change.
You can customize this behavior with the constructor option {sixelScrolling: false}
or with DECSET 80
(off, binary: \x1b [ ? 80 h
) and
DECRST 80
(on, binary: \x1b [ ? 80 l
) during runtime.
Cursor Positioning
If scrolling is set, the cursor will be placed at the first image column of the last image row (VT340 mode).
Other cursor positioning modes as used by xterm or mintty are not supported.
SIXEL Palette Handling
By default the addon limits the palette size to 256 registers (as demanded by the DEC specification).
The limit can be increased to a maximum of 4096 registers (via sixelPaletteLimit
).
The default palette is a mixture of VT340 colors (lower 16 registers), xterm colors (up to 256) and zeros (up to 4096).
There is no private/shared palette distinction, palette colors are always carried over from a previous sixel image.
Restoring the default palette size and colors is possible with XTSMGRAPHICS 1 ; 2
(binary: \x1b[?1;2S
).
It gets also restored automatically on RIS and DECSTR.
Other than on older terminals, the underlying SIXEL library applies colors immediately to individual pixels (printer mode), thus it is technically possible to use more colors in one image than the palette has color slots. This feature is called high-color in libsixel.
A terminal wide shared palette mode with late indexed coloring of the output is not supported, therefore palette animations cannot be used.
SIXEL Raster Attributes Handling
If raster attributes were found in the SIXEL data (level 2), the image will always be truncated to the given height/width extend. We deviate here from the specification on purpose, as it allows several processing optimizations. For level 1 SIXEL data without any raster attributes the image can freely grow in width and height up to the last data byte, which has a much higher processing penalty. In general encoding libraries should not create level 1 data anymore and should not produce pixel information beyond the announced height/width extend. Both is discouraged by the >30 years old specification.
Currently the SIXEL implementation of the addon does not take custom pixel sizes into account, a SIXEL pixel will map 1:1 to a screen pixel.
IIP Support (iTerm's Inline Image Protocol)
Set by default, change it with {iipSupport: true}
.
The IIP implementation has the following features / restrictions (sequence will silently fail for unmet conditions):
width=200%
).pixelLimit
(pre- and post resizing).iipSizeLimit
.The internal storage holds images up to storageLimit
(in MB, calculated as 4-channel RBGA unpacked, default 100 MB). Once hit images get evicted by FIFO rules. Furthermore images on the alternate buffer will always be erased on buffer changes.
The addon exposes two properties to interact with the storage limits at runtime:
storageLimit
storageUsage
By default the addon will show a placeholder pattern for evicted images that are still part
of the terminal (e.g. in the scrollback). The pattern can be deactivated by toggling showPlaceholder
.
The addon provides the following API endpoints to retrieve raw image data as canvas:
getImageAtBufferCell(x: number, y: number): HTMLCanvasElement | undefined
Returns the canvas containing the original image data (not resized) at the given buffer position.
The buffer position is the 0-based absolute index (including scrollback at top).
extractTileAtBufferCell(x: number, y: number): HTMLCanvasElement | undefined
Returns a canvas containing the actual single tile image data (maybe resized) at the given buffer position.
The buffer position is the 0-based absolute index (including scrollback at top).
Note that the canvas gets created and data copied over for every call, thus it is not suitable for performance critical actions.
The addon does most image processing in Javascript and therefore can occupy a rather big amount of memory. To get an idea where the memory gets eaten, lets look at the data flow and processing steps:
term.write
(terminal)term.write
might stock up incoming chunks. To circumvent this, you will need proper flow control (see xterm.js docs). Note that with image output it is more likely to run into this issue, as it can create lots of MBs in very short time.pixelLimit
(default 16M pixels), the decoder holds 2 pixel buffers at maximum during decoding (RGBA, ~128 MB for 16M pixels).storageLimit
is hit (default 128 MB). The storage holds a canvas with the original image, and may additionally hold resized versions of images after a font rescaling. Both are accounted in storageUsage
as a rough estimation of pixels x 4 channels.Following the steps above, a rough estimation of maximum memory usage by the addon can be calculated with these formulas (in bytes):
// storage alone
const storageBytes = storageUsage * storageLimit * 1024 * 1024;
// decoding alone
const decodingBytes = sixelSizeLimit + 2 * (pixelLimit * 4);
// totals
// inactive decoding
const totalInactive = storageBytes;
// active decoding
const totalActive = storageBytes + decodingBytes;
Note that browsers have offloading tricks for rarely touched memory segments, esp. storageBytes
might not directly translate into real memory usage. Usage peaks will happen during active decoding of multiple big images due to the need of 2 full pixel buffers at the same time, which cannot be offloaded. Thus you may want to keep an eye on pixelLimit
under limited memory conditions.
Further note that the formulas above do not respect the Javascript object's overhead. Compared to the raw buffer needs the book keeping by these objects is rather small (<<5%).
Why should I care about memory usage at all?
Well you don't have to, and it probably will just work fine with the addon defaults. But for bigger integrations, where much more data is held in the Javascript context (like multiple terminals on one page), it is likely to hit the engine's memory limit sooner or later under decoding and/or storage pressure.
How can I adjust the memory usage?
pixelLimit
storageLimit
storageUsage
you can do runtime checks and adjust the limit to your needs. If you have to lower the limit below the current usage, images will be removed in FIFO order and may turn into a placeholder in the terminal's scrollback (if showPlaceholder
is set). When adjusting keep in mind to leave enough room for memory peaking for decoding.sixelSizeLimit
mpv
with its SIXEL renderer (tested in the demo). For 3rd party xterm.js integrations this
furthermore depends highly on the overall incoming data throughput.