Retain terminal processes between window reloads

[sibowilldo]

Hi

Please implement Hot Exit on Integrated Terminal.

Thanks

• VSCode Version: 1.9.0
• OS Version: Windows 10

[Tyriar]

This would be cool, I'm not sure it could be done in the sense of retaining the processes (as they should be killed once VS Code is done). Maybe the shell and environment could be restored though?

There's a PR out for hyper which is relevant https://github.com/zeit/hyper/pull/945

[ppot]

@Tyriar Thanks. I'm here if you have questions!

[sibowilldo]

@Tyriar Yes restoration is exactly what i was referring to, e.g If I'm working on a Laravel Project and I run the serve artisan command, after restarting VS due to an update, it would be a great implementation if VS would automatically run the "php artisan serve" command on its own.

[Tyriar]

@fabiospampinato just published an extension which allows you to configure several terminals and launch them all at once! https://marketplace.visualstudio.com/items?itemName=fabiospampinato.vscode-terminals

[fabiospampinato]

Looks like my extension mitigates this issue almost entirely.

It would be pretty cool to retain the output of those terminals between sessions, but I guess an API for getting it as text, and another one for writing it (without executing it as a command) are necessary for this (@Tyriar do you think it would be possible to add them?) . We can leverage the fact that by using this extension no two terminals opened by it would probably have the same name, so we could safely link terminals' names with their text.

I guess it would also be nicer to have the terminals ready as soon as you open the project's folder, without the need to run the Terminals: Run command, I'll add an option for this tomorrow :+1:

[Tyriar]

@fabiospampinato no plan on adding the ability to read or write right now.

Something that will help you though is that you will be able to add/remove environment variable from the environment in 1.15.

[Tyriar]

This issue is a little ambiguous at the moment, here's what I see covered by it:

1. Support reloading VS Code without terminating terminal processes

This can be accomplished by moving the terminal process to live under Electron's main process instead of the browser process, then reinstating the terminals after a load occurs. There is quite a bit of refactoring that needs to happen here, probably most significantly is splitting node-pty into 2 components; one that launches the process and exposes the file descriptorunix socket/named pipe (under the main process), and one that takes the file descriptorunix socket/named pipe and connects to it (under the browser process). We do not want the pty's data stream to just be passed through IPC as it will slow down everything.

2. Restore terminals with their cwd/environments they were created with when restoring a workspace

Note that this should only happen when 1. does not occur. The solution for this should allow for split terminals to be restored in the future https://github.com/Microsoft/vscode/issues/7504

[jerch]

one that launches the process and exposes the file descriptor (under the main process), and one that takes the file descriptor and connects to it (under the browser process)

@Tyriar Some OSes do not allow the pty master fd to be duped/cloned, e.g. used by another process than the original creating process. IPC is the only platform independent way imho (os pipes are pretty fast though).

[Tyriar]

@jerch good to know. The sheer amount of data that's possible to go through the terminal is the concern as it could lock up the main process which is shared between all VS Code windows :confused:. Maybe I'm just overestimating the impact it would have on the main process? Any more details on the some OSes part?

[jerch]

@Tyriar It is still possible to carry the master fds around without duping it, if O_CLOEXEC is not set. It is important to reenable it in the browser process, otherwise the fds will leak to all subprocesses (assuming you have no control of electron's subprocess creation, the fds could also be closed by hand between fork and exec).

[Tyriar]

I amended my commend in https://github.com/Microsoft/vscode/issues/20013#issuecomment-340026972 to replace file descriptor with named pipe (Windows) and unix socket (Linux/macOS) as file descriptors are per-process. Implementation would use the net module https://nodejs.org/api/net.html#net_ipc_support

I'm thinking this is the way forward as node-pty is a standalone project (not just VS Code), plus plumbing terminal data streams through the standard main process IPC could block other communication if the terminal is spitting out lots of data.

[Tyriar]

I split this issue into 2, this issue will cover keeping processes between window reloads and https://github.com/Microsoft/vscode/issues/44302 will cover restoring the session layout and relaunching of processes.

[dahjelle]

Here's my issue (which I believe is on-topic for this issue instead of #44302 or another, but feel free to correct me):

I run my Node app from the command-line, and keep it open and running during development (to view error message output, logs, etc.), with a second terminal open for inputing terminal commands as I go. As it stands, if I want to install an extension, I need to reload the window, which kills my app.

tmux helps, as I can just re-attach to the tmux session, but that has it's own issues with non-native scrolling and somewhat strange text selection and whatnot.

It'd be fantastic if I could reload the window (or install/uninstall extensions) without losing the processes I have running in the VS Code terminal.

[fabiospampinato]

@dahjelle you could replace tmux with screen, you should find less issues with it, if you only need to re-attach to sessions.

[fabiospampinato]

I've just implemented something like this in Terminals Manager, check it out:

@Tyriar I had some problems developing this:

1. I get an empty terminal if I start the multiplexer while the pane is hidden, so I had to show it first.
2. If I start tmux and immediately after that send some commands to it, it doesn't get them, so I had to wait a bit.
3. Notice how in the gif above we start with 1 terminal but we end up with 2 of them https://github.com/Microsoft/vscode/issues/39137.

[Tyriar]

@fabiospampinato VS Code will eventually support this natively, I'd recommend against going down the tmux route as then it doesn't work for Windows. Also you can already get this by following this tutorial https://medium.com/@joaomoreno/persistent-terminal-sessions-in-vs-code-8fc469ed6b41

If I start tmux and immediately after that send some commands to it, it doesn't get them, so I had to wait a bit.

This is because you send the key strokes to bash/zsh, which eats them before tmux is launched.

[nikibobi]

Any updates on this? It has been a year since last comment. I'm also interested in saving terminal splitting layout and each terminal's current directory.

[Tyriar]

@nikibobi no update really, but we definitely want to support this for window reloads eventually. For setting up an initial state we're going to defer to extensions like Terminals Manager.

[mikemak-es]

I think i have a slightly similar issue to @dahjelle . When running an npm watch task and vscode quits for whatever reason, the terminal is terminated but the running node process is still there if i check macos activity monitor - sometimes this causes chaos when multiple npm run tasks are now running trying to write to the same files etc - and it takes a while to remember " oh yeah, vscode restarted and there are probably some lingering node process". If the terminal quits with vscode, is there a way to ensure spawned child processes are terminated with it or somehow reattach the process when vscode comes back?

[jerch]

@mikemak-es Thats weird - currently vscode closes the pty thus a SIGHUP should be sent to the slave processes which should terminate any terminal associated process gracefully. Can you check if the process also keeps running when started from Terminal.app or iTerm? (Maybe npm detaches some processes from the terminal, then the problem cannot be solved from vscode's side.)

[mikemak-es]

@jerch sorry for the late reply, indeed you're correct! What we were running did spawn some processes which were detached.

[dejunzhou]

On this topic, I suggest vs-code do something similar to Jupyter's terminal feature, where user can see the list of terminals currently open. These terminals won't be killed when user close the browser window. and these terminals can reopen or reattached. Refer to jupyter terminal manager related stuff for implementation details.

[Tyriar]

@dejunzhou the complexity here revolves around where the processes are spawned, for Jupyter you have a server which doesn't die when the browser window does so the problem is trivial. Here the processes currently live underneath the window process.

[dejunzhou]

@Tyriar I switched from Theia to VSCode, to be more accurate, to code-server. I admit I'm not aware of all the scenarios of vs-code is used. What I need is to setup and run the vscode on my dev server and I can log on from anywhere through web browser. After closed the browser, I can resume the session the next time when I reopen the same url. I expect to see the same set of files opened in the main UI and terminals previously opened still running.

Currently Theia does what I expected. It offers the UI similar to vscode. Are you saying this kind of scenario is not supported by vscode natively?

[Tyriar]

@dejunzhou feature requests for code-server are off topic

[yxiao-wish]

any updates on this?

[JavaCS3]

any updates on this?

The xterm-addon-serialize is intended to solve this problem. It's almost ready, but still takes some times.

[Tyriar]

Note that I don't mean to get people's hopes up on this as this was just an exploration. I put together a prototype while exploring https://github.com/microsoft/vscode/issues/74620 and got persistent terminals working in the prototype:

We can't use that solution as is and it will need a decent chunk of iteration time to build and polish, but it validates that it works 😄. It even seemed to work perfectly when running a long command and reloading.

The restoring of terminal content is done in the prototype by just keeping track of all the data events going to the terminal and replaying them. This is a slow, bad way of doing it but I was pleasantly surprised how fast even that seemed to work, I particularly like how the terminal is ready with colors as the editor is loaded without colors. The end solution for serializing the buffer will use the serialize addon that @JavaCS3 has been working on to send over minimal data to restore the terminal to it's expected state.

More details on the exploration in https://github.com/microsoft/vscode/issues/74620#issuecomment-642825176 if you're interested.

[jerch]

I particularly like how the terminal is ready with colors as the editor is loaded without colors.

Because you steal precious startup cycles from the editor? :thinking: - nah just kidding, looks really promising. This will be a nice usability boost for the terminal. :+1:

[Bessonov]

It's not just about the reload. I connect my vsc with a remote docker container. Everything except terminal stays after reconnect to the container. It would be a HUGE game changer for us, if there is no need to restart everything every day.

[jerch]

@Bessonov As intriguing your use case sounds, it also might be overstretching what will be possible with this. Note that a "terminal connection" is a limited active stateful resource held by the OS (as PTY/TTY or ConPty), which cannot easily be put away for later re-usage (much like a socket itself). To compare it with a socket - it would have to be re-established, but other than a socket with a stateless protocol, it does not have any semantics out of the box to do that, it would always lose the state and bubble its hang-up to the attached processes.

The consequence of this is - the underlying TTY resource has to be held all the time, not sure if that is planned to work remotely. This also might run into troubles, if you use tons of containered terminal connections, as they would stack up on the docker host (PTYs have strict limits in the OS). It also raises the question, who shall wipe stale PTYs (how to know, whether a certain PTY will be reused in the future).

Edit: A way to overcome this on the docker host in POSIX terms would be to write some sort of PTY/TTY driver with a terminal hosting service/demon, which allows remote terminals to re-attach easily and to orchestrate held TTYs. But thats a fairly complex goal, if it should work from any host system with different container solutions.

[Tyriar]

A way to overcome this on the docker host in POSIX terms would be to write some sort of PTY/TTY driver with a terminal hosting service/demon, which allows remote terminals to re-attach easily and to orchestrate held TTYs. But thats a fairly complex goal, if it should work from any host system with different container solutions.

@jerch this is kind of what I had in mind actually, for containers it would depend on whether the container shut down but the idea was to also run a node target version of xterm.js with the serialize addon in this daemon such that when the connection is re-established, we can also restore the terminal's state.

Not idea when I would get the time to actually execute on this though.

[mmis1000]

Is it possible to separate the implementation detail and interface for that. So relies on different implementation on different platform is possible?

The way to create a daemon different between systems, smash them in a single JS file seems would increase the difficulty hugely.

Maybe something like

{
  "terminal.persistent.linux": {
    "enabled": true,
    "launch": ["tmux", "-L", "$VSCODE_INSTANCE", "new-session", "-A", "-s", "$TERMINAL_INSTANCE", "$COMMANDLINE"],
    "list": ["tmux", "-L", "$VSCODE_INSTANCE", "list-session"],
    "kill": ["tmux", "-L", "$VSCODE_INSTANCE", "kill-session", "-t", "$TERMINAL_INSTANCE"],
    "killall": ["tmux", "-L", "$VSCODE_INSTANCE", "kill-server"]
  },
  "terminal.persistent.windows": {
    "enabled": true,
    "launch": ["..."],
    "list": ["..."],
    "kill": ["..."],
    "killall": ["..."]
  },
  "terminal.persistent.mac": {
    "enabled": false,
    "launch": ["..."],
    "list": ["..."],
    "kill": ["..."],
    "killall": ["..."]
  }
}

The vscode may ship one written with xterm.js/node-pty by default.

But if the way vscode bundled does not seems to work well with the environment user is using (something like remote container). The user can change it to something else.

[Tyriar]

@mmis1000 no configuration or tmux would be needed for my vision, it would just work magically across all platforms, probably with a setting that lets you set how long to keep the processes around without a VS Code window present.

[jerch]

@Tyriar Would be nice to have some standardized "terminal over ethernet" protocol in between (with all niftiness like replay, auth, read/write multiplexing, TTY orchestration, etc.), this way the daemons could run decoupled on several hosts and some aggregator logic in vscode just triggers those paths on reconnect. Just a wild idea :smiley_cat:

[mmis1000]

Em...A standardized terminal multiplexer protocol.

Something like language server protocol but for terminal? Actually define the protocol so we can have a shared way of handling terminals instead of every single terminal multiplexer use its own protocol, tmux use a, screen use b, whatever.

[jerch]

@mmis1000 Imho a standardized terminal multiplexer protocol is a slightly different story. What I meant above is more about to specify how the data can be provided on both ends (server/client) and to transport it securely. It is more about unifying the transport interface so custom daemons or aggregators can easily plug into it. How the transported data itself is shaped (thats what a full multiplexer protocol would address), is imho up to the daemon/aggregator in use.

Or in terms of tmux - currently ppl use ssh as transport to pair/duplicate to remote sessions, which itself is not disconnect tolerant. Here a more versatile protocol could come handy, that already offers typical TTY orchestration needs. Whether it will be adopted, ofc depends on its security offers (competing with ssh it a tough one :wink:) and ease of implementation/maintenance (a full stack HTTPS based protocol with rest API and JSON buckets prolly will be rejected by most multiplexer maintainers).

[mmis1000]

@jerch But connect back to the terminal without replaying everything is exactly the only thing a terminal multiplexer must do. Without such capacity. You will need to send every byte the process write to stdout over the Ethernet. And the IPC between vscode process, your network, the client is probably going to suffer.

In case of the ssh, such accident like cat xxx.mp4 is normally going to hang your client completely, the only thing you can do is close it and reopen a new one.

But as a terminal, the only thing you need is the last several lines (depend on your screen and back buffer size) that is enough to fit into your screen. You don't really care the rest because you can't see them anyway. You will want to just skip over the giant amount of data and show the last several line. Such capacity is actually impossible without some sort of terminal multiplexing.

BTW, the reason somebody complaining the scroll in tmux is due the how tmux handle back buffer. The tmux ate the back buffer completely and implement scroll on its own, resulting in the client is no longer able to mouse scroll the screen.

[jerch]

@mmcru Well the replay and the pairing ideas are just two small aspects of the much broader field of terminal multiplexing. Both prominent multiplexers screen and tmux solve that in their very own way with true emulation in between. Imho thats not the goal here. Ofc a well written protocol could go that deep into details, but it again would need screen and tmux implementing those details with a daemon and aggregator component. (Why should they - they already have their genuine abstraction for that.)

Imho the idea here is much more basic - as long as a terminal endpoint is attached, the data gets delivered right away (no to only a very small buffer on daemon side). Thats important for live sessions, as otherwise the latency will get annoying. There is no emulation in between. What I meant here as read/write multiplexing is something different - ideally the daemon knows how to create new readers or even writers (for shared read-only or read-write terminal sessions). This would allow easy "terminal collaboration", like attaching another vscode aggregator of person B to person A's session. Ofc this will need some security measures, otherwise it should never leave the testbed thinking sandbox. Why I nag about ssh above is - it is not disconnect tolerant, hard to setup right in cloud/container envs etc - thus it might be worth to think about its own security layer.

Beside that, there is no further multiplexing idea (esp. no notion about mapping multiple terminal views into one bigger one - the original multiplexing idea).

But connect back to the terminal without replaying everything is exactly the only thing a terminal multiplexer must do.

Why do you think so? A terminal recorder would have to do that, but is no multiplexer. Also a multiplexer could skip that feature (and most do, as they instead do in between emulation and just hook into that permanent background live session).

You will need to send every byte the process write to stdout over the Ethernet.

Thats always the case for attached sessions. There is no way around it, as terminal data is stateful and not layered/boxed by any means (every follow up state might depend on every previously activated state, even if it was set 10 days ago or happened 10GB of data earlier). Btw thats the real challenge of implementing a good working replay.

In case of the ssh, such accident like cat xxx.mp4 is normally going to hang your client completely, the only thing you can do is close it and reopen a new one.

If properly implemented, nope. If this happens with some stack of tooling - it is a hint, that some component does not deal well with flow control (then it might even segfault at some point).

But as a terminal, the only thing you need is the last several lines (depend on your screen and back buffer size) that is enough to fit into your screen.

Thats only what you think is needed as it is visible to you. A terminal emulator in fact holds much more state beside the last n-th buffer rows. Ofc the daemon would have to cut data in replay history somewhere to not get into storage nightmare, but it is not as simple as "save only the last n-th rows".

You will want to just skip over the giant amount of data and show the last several line. Such capacity is actually impossible without some sort of terminal multiplexing.

Not sure what you mean with that. Skipping over an existing stream of terminal data is possible without any multiplexer (in the narrow terminal multiplexer sense, in fact the daemon would be a multiplexer in the common sense yes), but it needs proper state parsing over the stream data. A replay then could be done by collecting up the initial state over the skipped parts, and start pumping stream data from that interesting point of data/time.

[mmis1000]

Ofc the daemon would have to cut data in replay history somewhere to not get into storage nightmare, but it is not as simple as "save only the last n-th rows".

That didn't work unless there is a reset control code somewhere in the response, then it can serve as a cut point.

Terminal is always stateful. Sentence after it write over the result of previous one. Cursor move relative to previous position. You can't get the correct result unless you actually replay all of it. Who replay all of does not matter, some may just split everything to client and let it do it, some do it at server and return the result of replay to client (the case of screen or tmux).

[jerch]

@mmis1000 Again not sure what you are trying to say, as it is somewhat self contra-dictionary:

That didn't work unless there is a reset control code somewhere in the response, then it can serve as a cut point.

True, but unreliable for a "skip-over" replay, as sequences like RIS or DECSTR typically dont happen during normal session run.

Still your observation here contradicts this:

You can't get the correct result unless you actually replay all of it.

If you find a state-destructive sequence (like RIS, and DECSTR to some degree), you dont have to replay it all. Note that for me there is a difference between full replay (as full emulation) vs. search accumulated data for for those points.

Again - thats not reliable, as it is likely to never happen (thus you are likely to never find a clean state entrypoint for replay beside the very beginning). But before replaying everything, there is another option you didnt consider - do some state back reporting from an active terminal endpoint, and link this to the stream data position. This way you get an unclean entrypoint (a replay from that point needs to apply the state itself first, before working on the follow-up stream data). Those unclean entrypoints can be managed by the daemon, even with discarding older stream data completely, while a healthy looking replay from there is still possible. No second full replay needed at all. (This is abit like doing terminal state snapshots, and starting from there.)

[mmis1000]

I'd consider assume you can always read data from client not reliable at all. As it always have been, client may lost/corrupt data for some reason, like a app crash, system BSOD. A recovery method that require client not encounter error at all is …weird? Even in the vscode case. If it is going to run in the browser, then things like such is very likely to happen.

[jerch]

@mmis1000 Sorry I cannot see how your last comment is related to anything above. Plz lets not jump to random conclusions without proper context, as it derails the discussion.

[mmis1000]

From the beginning, the only reason I think it is a terminal multiplexer protocol is the reconnect without resend everything', nothing more is included.

Terminal protocol is a stream of patch, and unlike video, you don't have a P frame somewhere by default to skip over to near the end of stream unless user issue a 'reset' manually (which is basically does not exist and does not matter at all because user seldom do that).

In order to able skip over without break the output completely. Somewhere in the system need to generate the P frame. So you can 'start from here' or you just can't (think of a video that trying to start from a I frame, it really can't).

It can be client (the terminal send all its last state to server, and the server response with it when reconnect), or it can be server (something like tmux but with far less function). Or probably mixed, server only do it if client is unavailable.

But I feel completely rely on client and assume the client will just do it well is a bad assumption, clients always break for variable of reasons. I am not not think of it. I am assume it is going to break. So the only reliable option will be the server render it (always or on required).

Is there a third way can handle the reconnect correctly without show you a half broken screen? Or show you a non broken screen on reconnect isn't one of the goal?

[jerch]

@mmis1000

There are different scenarios possible to get better fault tolerance (imho thats what you are trying to say above). But note that currently a short terminal disconnect/reconnect is simply not possible for vscode by any means, thus even a best effort approach, which has to bail out for rare edge cases, is already a huge overall win.

What you describe as "it is going to break" is currently the default case from where developing a better idea would start. Whether the final concept should aim for 99% fault tolerance or is good enough to cover most cases (80:20 rule), depends alot on the circumstances, like manpower to implement such a solution and last but not least its technical implications like additional runtime costs (more CPU/memory hungry for "play along emulation", storage needs on server side etc). Since I am not going to implement it, I wont judge about it. From a user's perspective anything into that direction will be better than now, as easy as that.

If you have some profound terminal interface experience and time to share, maybe try helping to get it shaped. I for myself have not enough time for that, thus can only give some ideas here.

[Bessonov]

Usually I close all terminals before exit because of "loss" of terminal tabs. But yesterday I didn't. Today I've noticed, that after I've connected to remote container my terminal was restored. It lost terminal tabs, but scroll history is here. It work even with multiple terminals. And it is able to processing executed commands. For example, I can execute sleep 60; echo "test", quit and connect again. It seems like it doesn't lost running command and after 60 seconds pass, it prints "test". Tested it with simulation of network failure and "reload window". It worked as expected.

For me, everything is already here. I'm very happy :D

(the only one wish - terminal tabs should be restored too or integrated)

Version: 1.51.0 Commit: fcac248b077b55bae4ba5bab613fd6e9156c2f0c Date: 2020-11-05T18:16:10.374Z Electron: 9.3.3 Chrome: 83.0.4103.122 Node.js: 12.14.1 V8: 8.3.110.13-electron.0 OS: Linux x64 5.4.0-56-generic

[Bessonov]

Unfortunately, it doesn't work every time. It seems like something dying after some time...

[Tyriar]

Done in https://github.com/microsoft/vscode/pull/116449 🎉