Scenario: Add support for 3rd-party extensions

[zadjii-msft]

Updated thread from #555

Now that we have a better understanding of what work will be needed to support 3rd party extensions ("plugins"), this thread is the megathread to track all the work that's needed to support them.

Before any real work can begin here, we're waiting on some support from the Windows operating system that will be coming Soon^TM. Fundamentally, third-party code extensions for packaged applications (like us) isn't supported currently.

This is not work slated for 1.0. It may not even land in 2.0. This work will likely span multiple releases. Whenever this work does end up starting formally, we'll likely have a number of minor versions with prototypes of the extensibility model, with breaking changes up until the major version release of the Terminal. So if extensions are scheduled for 3.0, then 2.4 might introduce bts of the extensibility model that are later broken in 2.5, 2.6, etc.

What should extensions be able to do?

• Add their own profiles via Dynamic Profile Generators
• Customize the rendering of the Terminal, with things like:
  • custom shaders, font renderers
  • custom formatting of the buffer (like hyperlinks)
• Add their own Color Schemes, keymaps
• Modify the UI theme (for adjusting appearance of tabs, scrollbars, etc.)
• Modify the layout of the UI
  • Add elements to the UI like status bars, toolbars, minimaps
  • reposition existing elements of the UI? Tabs on bottom for example.
• Create Panes with their own UI elements
• Read the contents of the text buffer, for parsing things like hyperlinks
• When extensions modify the UI, they'll need to adjust how large the initial window size is
• Add commands to the Command Palette, including nested commands
• Add commands to various context menus (such as the TermControl context menu, TabViewItem menu).

Work needed to support extensions

Some of this work is more directly related to extensions, while others on the list are simply adding features to the terminal to provide points for extensibility in the future.

• [x] Make Profile, ColorScheme, GlobalAppSettings proper WinRT types #3998
• [ ] Make Tab, Pane proper WinRT types #3999, #3922
• [x] Support Panes with non-terminal content #997 (spec: #1080)
• [x] Add an optional right-click menu to the TermControl (#3337)
• [x] Add the command palette #2046 (spec: #2193)
• [x] Enable nested commands in the command palette #3994
• [ ] Add comprehensive "XAML" theming functionality #3327
• [ ] Design Extensibility model - this covers actually figuring out the API surface by which developers can write extensions for. This is most of the work
  • This will involve developing a couple extensions in-house and partnering with some community members to explore extensibility model

Mega-list of extension ideas

Dynamic Profiles / settings

• 3821 Add Developer PowerShell and Developer Command Prompt Generator

• 1394 Upon installation, add a Git Bash profile, if Git for Windows is present

• 1280 Feature Request SSH-Telnet-Serialport connection

• 5049 Device Portal

• 5900 Include a profile that will connect to Visual Studio Codespaces

• 6105 Add WSL distro default desktop wallpapers as default background

• 8339 "ask parameter" for profile. profile with variable that is prompted-for

• 9031 Auto generate profile from .ssh/config

• 10943 Importing PuTTY sessions

• 11641

  • This isn't "dynamic profiles", it's more "commands that are provided to the command palette via an API NOT a plaintext file"

• 12773

Additional Connection types

• 1999 Plugin: add support for [XYZ]MODEM file transfers

• 694 Suggestion: "One-click & snap" connect to bluetooth/serial devices and network hosts using QR Code and Code 128

• 3656 Add support for tmux Control Mode

• 5321 Support HTM (headless terminal multiplexer) for remote pane/tab management

• 11064 Bastion shell similar to azure cloud shell

• 13245

Buffer Parsing/Manipulation

• 3196 Feature Request: Smart Double-click Selection (regeces?)

• 2671 Feature Request: link generation for files + other data types

• 574 Design Clickable Links & Link preview feature/extension

• 5226 Parenthesis matching in text

• 6297 Highlighting matches in all history

• 5916 Triggers(Including Text Strings) and Actions (internal or external calls)

• 6632 ITerm2-like terminal autocomplete

• 7749 ability to pipe scrollback buffer to a pager or temrinal editor

Suggestions

• 17344

  • that pretty much tracks all of it.

UI Elements

App elements

• 3459 Add an optional status bar to the bottom

• #2994 Feature Request: allow to pick the color of the tabs
  • We actually did this ourselves!

• 2934 Feature Request: Icon buttons to start relevant shell types

  • (Maybe something like a toolbar? See also #4531, #5273, #7207)

• 1595 Feature Request - Scripts Panel

  • A panel for saved commands #11270

  • 5273 Favorites/Shortcuts/Quick List

  • Also maybe implies some sort of integration of storing their own settings that are linked to Terminal profiles?

• 2601 Feature Request: optional splitting of view into scroll pane and static pane while scrolling

• #1502 Feature Request: An advanced tab switcher
  • We actually did this ourselves!

• 835 Feature request: Enable customization for tabs on bottom

• 644 Feature Request: /help commands output to collapsible pane

• #4444 It would be nice to have a kind of popup with a tab name when you switch tabs with CTRL+TAB in fullscreen mode.
  • We actually did this ourselves!
  • Let's add a setting for this as well, this sounds like a not so terrible idea.

• 5426 "Peek" button to show screen behind Terminal

• 5636 Add support for Azure Cloud Shell code Editor

  • In this case, a particular connection might need to be able to spawn a new pane, with HTML content in it

• 16387

  • We closed this cause we're so very unlikely to do it, but it's not a bad idea post-#835

• 16420

  • "Extensions should be able to add settings UI elements"
• more...

Control Elements

• 3121 Enhance shell autocompletion with a cool new user interface and shell completion

• 2471 Feature Request: ⬇️ button to appear when scrolled many pages up

• 2226 Feature Request: Scrollable map view for each tab [minimap]

• 1527 FR: IDE-style marks on scrollbar

• 5278 Provide a 'Paste mode' to allow a paste keybinding to be 'smart'.

  • A single keybinding that lets the user chose between "paste", "send ^V", or "Cancel".
  • Might require adding a custom argument to a ShortcutAction we've defined? Or its own ShortcutAction.

• 6025 FPS overlay for performance debugging

• 6511 Control how multi-click selection operates

• 6979 Have text as the background instead of just an image

• mjpeg streaming for terminal control background #6242

• 6497

• 15837

  • similar to the above
• Display a indicator on the cursor's line #9993

• 10175

  • Hey we did this in 1.19!
• rainbow cursor? #10442
• Show and allow editing of typeahead #10690

• 15104

Other Elements

• 5591 Opening the windows terminal inside Windows Explorer rather than from windows Explorer

• 6028 Background image for tab, not per split pane

Custom Rendering

• 3520 Feature request: 'age' output visually

• 781 Glow text exploration

• 7013 Allow background image to be pixel/fragment shader specified by me

• 7380 Add built-in D2d effects

• 9221 Smooth cursor animation/transition when moving

Advanced Settings

• 3900 Rolling background image like Windows Themes

• 7906 Change tab colour automatically based on tab title text using preset regex in settings

Miscellaneous

• 2516 Feature Request - Live Share support

• 469 Feature request: Recording

  • GIF Capture tool for the Windows Terminal #8098

• 5434 Voice Command

• 6372 Notification after a long running command finishes

• 7279 Randomize background images from an array of filesystem paths

• 8647 Auto display readme`s

• 8797 Extension: Pipe output to command palette in Terminal

• More elaborate SSH features (like PuTTY)
  • See: #4653, #11034

---

@ future me, 10/14/2021

I tried this in dev/migrie/fhl/adaptive-card-extension, but ran into a number of unfixable build issues that made me hate the world. I couldn't get another package built to be able to load that into the terminal at all. Maybe next month I'll try again. For now, it's on to honks.

[WSLUser]

You should add https://github.com/microsoft/terminal/pull/3468 that needs to be brought into an extension instead of living in the main code base.

[ntindle]

I would also appreciate being able to install these plugins from something like winget

[gbritton1]

Why on earth would you need support from the WIndows team? That doesn't apply to VS code, e.g. in any environment. Can't we just do it the same way?

[DHowett]

@gbritton1 Terminal is (for better or for worse) deployed as a "MSIX" package using the new Windows packaging/installer infrastructure; VSCode is not. That infrastructure is missing some features that would allow for:

• "In-process" extensions, which...
• ...support being updated while they're in use, and...
• ...can be installed for multiple users trivially.

Had we chosen to install using MSI, traditionally, into C:\Program Files... yeah, we wouldn't need anything else from the package subsystem. We'd also lose(1) support for easy, atomic updates for the entire Terminal. 🤷🏻

We've since learned that there's a way forward where we don't need in-process extensions, but we do want to make sure that it will perform adequately for things that impact the rendering engine or the text buffer.

(1): "Lose" insofar as it is easy. We could always write a bunch of code to support it, but on the balance we decided to invest most of our effort in making the Terminal and Console Host itself better. :)

[gbritton1]

Would it be simpler to change the packaging?

[ntindle]

I'm glad you put this out there. I honestly had the same opinion on why it required updates on the windows side. Thanks for clarifying that.

[zadjii-msft]

Notes comment:

See https://github.com/microsoft/terminal/tree/dev/migrie/fhl/dyndep.

https://github.com/microsoft/WindowsAppSDK/issues/66

Todo

• [x] Find 3p app using extension catalog
• [x] Register it as a dependency using dyndep
• [x] Be able to instantiate a WinRT class implemented in that package which implements an interface we declare.
• [ ] Figure out why the SampleApp package always wants to include MUX 2.7 in its package. Just manually deleting its activatableClasses from the Appxmanifest.xml worked but, sheesh
  • I didn't hit this in my public/terminal clone, only the private/Openconsole one. So maybe it's just a dirty repo.
• [x] Actually communicate the class name in the properties of the extension implementation
• [x] Create some sort of "smart dyndep scope" class / struct, to keep the package dependency alive...
  • [ ] ... and release the dependency when no longer needed
• [x] Try to get the extension to return a XAML object, created in codebehind
  • 🎉 This is like a pretty big milestone IMO 🎉
• [ ] Try to get the extension to return a XAML object, as defined in XAML
  • Possibly impossible: https://github.com/microsoft/microsoft-ui-xaml/issues/6299
  • 950311c
• [ ] Try to get the extension to implement a "HackerTyperConnection : ITerminalConnection"
• [ ] Clean up the way the extension projects build. Right now they're all bog-standard so they scatter shit all over the build tree. use the common props instead.
• [ ] Can an extension that creates a XAML button that gets added to the UI tree then somehow iterate upwards and get at the XamlRoot (and therefore, the whole Terminal UI tree?)
  • [ ] Or, acutely, just get at the XAML root even without user interaction like that?
• [ ] C#?
  • This was pain
• [ ] more...

---

Notes circa May 2023 Hackathon

• DynDep works perfectly.
• You can totally have an extension host instantiate classes that are hosted in another package.
  • You absolutely need to make sure they are declared as activatableClasses in the extension's manifest. This will only happen if you have a separate component (dll) that implements it. Hence ExtensionHost and ExtensionComponent. Component actually does the hard work.
• Extensions can build WinUI types at runtime, and add them to the host's UI: e0c8135
• You can even have the extension return a WebView2! That's awesome. 497fea8
• You need to make sure to keep a reference to the extension class, obviously, but even WinRT events work too: 82dc26f
• C# might be Hard to do. I don't think I understand how it works with WinRT anymore. I was able to convince the tooling that my class implemented the interface from the C++ project, but I don't know how to package it up to make it callable. And IIRC the new C#/WinRT stuff is all WASDK which is all WUI3 which is incompatible with WUI2.

---

Now, the questions are:

• How do we feel about in proc extensions?
  • Kinda a reliability nightmare. They crash? we crash.
  • Their code is literally running in our app context. https://github.com/microsoft/WindowsAppSDK/issues/66#issuecomment-1547015667.
  • Literally no security guarantees anymore. They can do whatever they want.
• What do we want to expose as extensibility points?
  • Additional actions
  • Additional profiles / dynamic profile sources
  • Send actions to the Terminal
  • Custom actions
  • Custom panes / UI surfaces
  • Custom suggestion sources
  • Custom connection types

[jonwis]

@zadjii-msft before you merge this we should chat. See my comment here - https://github.com/microsoft/WindowsAppSDK/issues/66#issuecomment-1547015667

[zadjii-msft]

@jonwis haha don't worry - that was firmly FHL code, not real shipping code. I use FHL to push the boundaries of what the platform will let me do. But I'm absolutely going to make sure there's proper boundaries set up here before doing anything like this in real Terminal code 😆

[jonwis]

(Also, don't get me wrong - I love it when we figure out the power latent in the platform. We've done sooo much here over the years that compiles together well but we don't have enough time or hands to document all the possible combinations. :) )