Overlay v2 - Githubissues

Motivation

The current implementation of the overlay has some issues:

There are complaints that the overlay button is too intrusive, especially in adventure mode where it covers important text.
It currently has to interpose every single screen in DF, which leads to a messy implementation and maintenance toil when new DF screens are introduced. This also prevents the overlay from functioning on DFHack-owned screens.
It currently can only have one widget and one action associated with left clicking on it, which makes integration with the upcoming control panel or widgets owned by other plugins/scripts (including 3rd party scripts) difficult.

This proposal seeks to solve these shortcomings. In addition, by supplying an overlay framework that other plugins/scripts can use to display information and respond to hotkeys, we can simplify many other DFHack tools and make it easier for new tools to supply viewscreen-bound functionality. Among other things, this solves the issue where scripts that want to overlay information over an existing screen have to add a new viewscreen to the stack, potentially breaking keybindings attached to that screen.

UX

The "[ DFHack Launcher ]" overlay button can be reduced to a 2x2 hotspot (in the upper left corner by default, but the position will be user-configurable). The hotspot could have a rendered image on some screens (title screen, main map screens, etc) and be invisible on others, but will always be reactive as a hotspot. Once we know more about how the steam renderer works, this can be changed to always rendering a semi-transparent DFHack icon, but we'll take what we can get for this version.

When the mouse rolls over that tile (or Ctrl-Shift-C is hit), a widget grows to show the tools whose hotkeys are associated with the current screen. This list will always include the launcher and (when it's ready) the control panel since their hotkeys are global. The hotkey for the menu itself would be filtered out.

The list will be scrollable with the mouse and keyboard and a short description of the selected tool (selected either by up/down arrow keys or by hovering the mouse cursor over the tool name) will be shown in an adjacent text box.

When the player clicks on a tool, hits enter, or hits a hotkey listed next to one of the tools, that tool is invoked and the widget disappears.

If the player moves the mouse from inside the widget area to outside the widget area, hits the menu hotkey, or presses Esc, then the menu closes without running a tool. If the mouse is still within the hotspot area, it will be ignored until it moves.

On the DF title screen we can display some text indicating where the hotspot is and how to use it. Something like:

<--- The DFHack menu hotspot is here. Move the mouse cursor to this spot or use the Ctrl-Shift-C keybinding to show it.

Overlay implementation

Overlay will be a framework that will manage overlay widgets provided by other tools (including 3rd party tools). A widget is registered with overlay via a json metadata file in the dfhack-config/overlay/ directory (see a few sections down for the metadata file format).

Overlay will provide two primary services:

managing hotspots that are always available (not bound to any viewscreen)
rendering widgets that are tied to a specific viewscreen

The unbound widgets will never have their render() or onInput() methods called they will not be interposed into any viewscreen's context. Even though we have access to the raw renderer, hooking into the raw renderer without interposing a screen's render() function is too risky at this point (that is, it's not well understood or trusted, doesn't play well with other renderers that want to hook, etc.). Widgets that are tied to a particular viewscreen will be interposed for render and feed when those viewscreens are active.

Overlay will track the mouse in its plugin_onupdate() and notify widgets when the mouse enters/leaves their bounds. The widgets can render differently on hover (become larger, display different information, etc.) or even spawn Screens to take over the viewscreen stack if they want to (the menu will do this to gain access to user input). See the Widget API section below for more details.

Menu widget

The menu widget metadata will be defined in dfhack-config/overlay/hotkeys_menu.json and widget code in plugins/lua/hotkeys.lua. The menu will be associated with the hotkeys plugin since it basically does the same thing, just in widget form. It will use all the same techniques and algorithms as the hotkeys plugin. It seems to make sense to pair them.

When the mouse enters the hotspot, the menu widget will appear. It can animate when it is shown to evoke the sense that it is growing from the hotspot. I found a public domain easing library we can use: https://luapower.com/easing The light reading I've done indicates that a ~100ms animation should be appropriate so that it smooths the appearance of the widget without feeling jarring nor time-wasting.

We want to optimize for opening gui/launcher, so that item should appear under the cursor when the menu appears. Since the widget is repositionable, we'll need to place the list of tools dynamically so the launcher item always appears under the mouse cursor. We might have to add the items in reverse and start at the bottom of the list if the widget is too close to the bottom of the screen.

The Lua widget will query its parent hotkeys plugin for relevant keybindings and commands to populate its list, and will use the existing technique (and code) for launching context-specific commands by their hotkeys, even when the widget's DFHack Lua screen is the top viewscreen:

Scan for relevant keybindings and commands before showing the menu viewscreen
Register those keybindings with the overlay menu's viewscreen (dfhack/lua/overlay_menu) with a command of the form overlay menuselect <index>
when overlay receives the menuselect command, it will dismiss the menu Screen and trigger the original command

hotkeys doesn't clean up registered keybindings, and it might be hard to do so. Even if we implement a "clear keybindings by command", there's not an easy way to determine if the player has set a similar keybinding that we would erroneously clear. It might be possible to get a token back from the keybinding set command that we could use to clear it later, but the benefit may not be worth it.

Widget config

Each widget will be defined in a different json file in dfhack-config/overlay/ so new widgets can be added easily, including by 3rd party tools. The dwarfmonitor widgets can move to this framework (with their backend logic staying in the dwarfmonitor plugin).

The name of the widget will be the name of the json file (without the .json extension) to ensure there are no naming conflicts. The suggested naming scheme will be <pluginorscriptname>_<identifier>.

The json file will keep configuration and state:

whether the widget is enabled
the screen position of the widget (relative to any edge)
the name of the plugin/script that provides the widget implementation (so we can require or reqscript it, depending on type)
the name of the widget class in that provider's environment
whether to monitor the widget boundaries as a hotspot on all screens, even if it's not rendered on that screen
which viewscreens the widget should be rendered on and receive the input feed for. could be empty if the widget is just a hotspot.

Widget API

Overlay widgets must inherit from overlay.OverlayWidget. The frame attribute will be set upon init to position the widget on the screen. The overlay widget can modify self.frame.w and self.frame.h at any time (in init() or in any of the callbacks) to indicate a new size. Overlay widgets can override the overlay_onupdate_max_freq_seconds attribute (default value: 5.0) if they want to have their overlay_onupdate() callback called more (or less) frequently. Set it to 0 to get called at the maximum rate. Be aware that running more often than you really need to will impact game FPS, especially if your widget is bound to the main map screen.

Overlay will instantiate and own the widgets. The instantiated widgets must not be added as subviews to any other View.

A widget class can define the following functions for integration with the overlay framework:

bool = overlay_onupdate(viewscreen) if defined, will be called on every viewscreen logic() execution, but no more frequently than what is specified in the overlay_onupdate_max_freq_seconds class attribute. Widgets that need to update their state according to game changes can do it here. The viewscreen parameter is the viewscreen that this widget is attached to at the moment. For hotspot widgets, viewscreen will be nil. Returns whether overlay should subsequently call the widget's overlay_trigger() function.
screen = overlay_trigger() if defined, will be called when the overlay_onupdate callback returns true or when the player uses the CLI (or a keybinding calling the CLI) to trigger the widget. must return either nil or the Screen object that the widget code has allocated, shown, and now owns. Overlay widgets will receive no callbacks until the returned screen is dismissed. Unbound hotspot widgets must allocate a Screen if they want to react to the onInput() feed or be rendered. The widgets owned by the overlay must not be attached to that new screen, but the returned screen can instantiate and configure new views.

overlay_onupdate() will always get called for hotspots. Un-hotspotted widgets bound to particular viewscreens only get callbacks called when the relevant functions of the viewscreen are called (that is, the widget will be rendered when that viewscreen's render() function is run; the widget will get its onInput(keys) function called when the viewscreen's feed() function is run; the overlay_onupdate(viewscreen) function is called when that viewscren's logic() function is run).

Performance considerations

The overlay will be interposing every DF viewscreen and calling out to Lua on every call to logic(), render(), and feed(). We need to be aware of processing times so we don't cause fps drain.

We will measure overlay processing times with high resolution timers and report SLO violations as warnings in the debug log. More detailed performance statistics will be accessible via some CLI command.

The goal is to avoid noticeably changing display and response latency at 100 FPS on average.

According to a handy online reference, 1 frame at 100fps is 10ms (yay math!).

So let's say initial targets (aggregated across all widgets) are:

additional per-render latency: <0.1ms avg, <1ms max
additional per-logic latency: <1ms avg, <50ms max
additional per-feed latency: <100ms avg, <5s max

If we find ourselves going over these limits, then players are being negatively impacted and we need to make changes. We can tune how much processing is done per render/logic and how often that processing is done in offending widgets by adjusting the overlay_onupdate_max_freq_seconds attribute.

To get per-widget performance characteristics, we can potentially use the chronos Lua library (MIT license). If that turns out to be infeasible, we can measure from C++, but we may not be able to get per-widget timings (unless we call into lua individually for each widget, which may come with its own performance penalty).

Overlay CLI

overlay (as a command) will provide the following subcommands:

trigger <widget name> (only if the widget implements the overlay_trigger() function and its associated screen (if any) is currently on top)
list (also indicates which can be triggered and which can be triggered right now)
enable|disable <widget name> (persisted to disk)
reposition <widget name> <screen position> (persisted to disk)
stealth to temporarily not render the widgets until the next screen transition (triggered widgets can still render)

Overlay GUI config

gui/overlay can be the config screen for the overlay plugin. It shows the list of widgets and allows them to be enabled/disabled. As a widget is selected in the list, its on-screen position will be highlighted. Enabled widgets can be clicked and dragged around the screen to reposition them, or a "reposition" button can be selected (via mouse or keyboard) that will allow a widget to be moved with the keyboard arrow keys (enter to confirm, esc to cancel). Widgets will be rendered, but will not be sent callback events while the gui config screen is active. Invisible hotspots will have a representative outline rendered. The json files will be updated accordingly.

DFHack / dfhack

Overlay v2 #2339