screenshots, diffs to annotate scrollback/history of fullscreen (tui/curses) apps

[elviskahoro]

from #78

I was just using htop for the first time in a while, and I had a thought that it would be interesting to extend (or merely apply) the notion of excerpting like I describe in #23 (3) to such apps (fullscreen / TUI / curses).

Status Quo

Currently when you use such an app (htop, vim, less, etc.) in Warp, once you exit it - the history block is rendered showing only the command

This is of very limited use to have in either the immediate scrollback of an active session, or in a retained artifact of a past session

(either: as I imagine in #23, something where you save a (shared) session to use as part of an incident post-mortem learning review or perhaps an educational tutorial / screencast, akin to https://asciinema.org et al ; or perhaps in the persistent cloud history that I believe is more solidly on the roadmap #33 )

What would be more useful?

What comes to my mind is some concept of capturing one or more "screenshots" of the app as it's running. Another idea (more suitable for editors) is a diff of a file before and after invocation (at least for file(s) which are passed as command line argument(s) to an editor, rather than opened from within it).

Mocking that up, I imagine something like: screenshots-mockup.psd.zip

(For less and other pagers, it might also just be good to include a scrollable fixed-height buffer of the file contents; very much like what is currently shown in the share modal; or just a file:/// hyperlink?)

How?

These "screenshot"s could either:

1. render the contents of the ~frame buffer captured as it appeared on screen- capturing text as text (so copy and pasteable, etc.), ansi escapes as is (raw or post processed);

   this might be easy enough with the current Data model, since I see that https://docs.rs/alacritty_terminal/0.14.0/alacritty_terminal/grid/struct.Grid.html#impl-Serialize and https://docs.rs/alacritty_terminal/0.14.0/alacritty_terminal/grid/struct.Grid.html#impl-Deserialize%3C%27de%3E

   this would creating a richer, likely smaller and more compressible artifact – and one more alike to what is currently offered with sharable command output; but could require more work

   and*/or

2. capture a raw rasterized screenshot of the window (in macOS, at least, I believe that capturing your own window this way does not require screen recording permission, but I might be wrong)- probably as a png.

*it likely would make sense to do (1) but e.g. have rasterized thumbnails

When?

It might make sense to make such capture at one or more of:

1. Application exit time. I think of this like what you see in the iOS multi-tasking view.

   Since each block already has its own grid that is being discarded(?) at precmd time, you would just need to serialize rather than discard that (as I guess you already do for non-fullscreen apps)

2. Periodically. I think of this like macOS's Time Machine visualization. This would be triggered either by some clock time interval, or heuristics like:
   • snooping on file saves (fd flushes of the exec'd process? or just plain old File System Events API / inotify ; for at least files mentioned in invoking command?)
   • a heuristic on how much of the screen is redrawn at once (between keystrokes)
   • network traffic?

3. (Only or additionally) by user request—via command palette, keyboard shortcut

   or maybe even a button-on-hover; possibly a menu option under a version of the ⋮ like we see in scrollback blocks, but just floated in upper right while a fullscreen app is active.

   When thus HCI-initiated it might make sense also to prompt for a caption/label (to indicate what is intended to be highlighted/captured), crop(?), and maybe even to (at screenshot time or after-the-fact) allow further annotations (something akin to, or identically (if rasterizing): macOS markup / Skitch / CleanShotX's image editor )

[elviskahoro]

from #78 Or, of course, you could fully do the https://asciinema.org thing and capture a ~scrubbable "video" of the whole fullscreen session (or a portion of the session started/stopped/paused/resumed by user input)

This one seems like it is likely to blow up fairly quickly in required storage space*.

It also, like the above (and also existing share functionality) but moreso, introduces opportunities for secrets to be inadvertently "caught on film"

– something for which I imagine it would be preferable to provide a redaction capability vs being all or nothing (keep or discard the recording; though it may be reasonable to just discard whole "frames" of the "video")

---

*though if you are or are going to be capturing/serializing grid ~deltas (or raw PTY events) to support either:

1. pushing changes in a collaborative session CRDT / (kinda) mosh style to fat clients (vs just running a thin client to mirror an authoritative state screen at a time)
2. playback of whole sessions (like Teleport offers: https://goteleport.com/teleport/features/record-ssh-sessions/ ) for audit or education

than you may already end up where you need to be to ~optimally store "videos" more efficiently than a series of individual "screenshots"

[elviskahoro]

From #2243 via @nachoaIvarez

Discord username (optional)

No response

Describe the solution you'd like?

It would be great to be able to get beautiful block screenshots, with a nice alpha and so on.

Is your feature request related to a problem? Please describe.

There are plenty of times when you want to share the output of a command, in the context of the terminal (not only plain text logs). Build output, reproducibility/hashing proofs, errors, and more are use cases.

Without this feature the workflow usually consists of clearing + resizing the console — and if you forget you usually end up cancelling and re-running on a cleared + resized console 🫠

I'm sure we've all done this in the past. I personally do this multiple times a week.

Warp already has the presentational block logic, this feature is right around the corner, almost for free!

Additional context

No response

How important is this feature to you?

4

Warp Internal (ignore) - linear-label:770f6576-d6c0-4e4f-a259-fc64b5156087

No response