node-dev / restore functionality

[danschumann]

Hey nice useful project here.

I noticed set -g @resurrect-processes '"~node-dev"' doesn't work, but i think set -g @resurrect-processes '"~node"' did restore, but restored only a blank node session, not my original node-dev index.js command.

I think node-dev actually calls node and the process runs as node.

What if tmux-restore detected whether a window had a process running, and you based restoring on the command that caused that program to run?

If I add node to my processes, i get a terminal that runs as if I had merely called node by itself, and it appears not to care what initially spawned the process.

This could be possible, right, by sniffing the history? I see the list of processes as a whitelist, interested by the currently running processes during a save. If they match, then run the same command again. Right?

[bruno-]

Hi, based on the project open issues I have to admit current implementation of process restoring does not play well with some programs.

I think node-dev actually calls node and the process runs as node.

Yea, I'm guessing that's the case. Please paste the content of the file symlinked to ~/.tmux/resurrect/last and we can verify this.

What if tmux-restore detected whether a window had a process running, and you based restoring on the command that caused that program to run?

I'd love to do that, but I'm not sure how to 100% reliably do that! (also, I'm not even sure if that's possible).

If you (or anyone else for that matter) can tell/show how to do this, I'll implement it in the plugin right away. This post explains how process command is fetched currently.

This could be possible, right, by sniffing the history?

I hope this is explained above.

I see the list of processes as a whitelist, interested by the currently running processes during a save. If they match, then run the same command again. Right?

Yep, this is how things work.

So, currently I don't know how to work this issue. I'll just label it as 'help wanted'. Any input from the community is welcome.

[danschumann]

Well before I can get too into this, when I say sniffing history, I mean typing history into the terminal, and it shows every command that was ran. If you did some sort of text search or regex on the last line, that would repeat the last thing they did in this window.

Are you currently detecting which windows have a process running, and can you run commands in that windows without breaking the process, i.e. history ?

[bruno-]

Hey - yep, I get what you mean about the history command, but I'm not sure if we can use that (reliably) to get the last pane/window command. Also, we don't want a bunch of history commands all around in tmux when resurrect saves. Users just want to save the state and continue working in the same environment (at least I want that).

As mentioned above, I'd be interested to know if you can make this work (non intrusively)! I'd be happy to do the additional work on implementing this in the plugin.

In short, this is the current process for all the panes being saved:

• get the tmux pane pid with tmux list-panes -F "#{pane_pid}". This is a parent pid for any process running in that pane/window.
• pgrep -lf -P <pane pid> to get the command (with arguments) currently running in the pane

[danschumann]

Okay.. soo.. I've got it.. ish

so, I got the pid of bash, similar to how you're doing it.

dan$ tmux list-panes -F "#{pane_pid}"
25731
dan$ DEBUG=* node-dev index.js arg2test

here i start the server. in your case, the server would be running already by the time you get the bash pid. the part that will be different is the pane_full_command

so, with the pane pid. (pane_full_command replacement -ish)

dan$ pgrep -lf -P 25731
19835 node
dan$ cat /proc/19835/cmdline | tr \\0 ' '
node /usr/local/bin/node-dev index.js arg2test

so you can see that /proc/#{process pid}/cmdline gives you the string that was ran to start the file, in this case its not identical, but it is functionally identical ( there may be programs that don't work like this, but for my case this is enough ). | tr \\0 ' ' is simply to replace the system delimiter with a space

dan$ cat /proc/19835/environ | tr \\0 \\n
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
DEBUG=*
EDITOR=vim
SHLVL=2
HOME=/home/dschumann
XDG_RUNTIME_DIR=/run/user/1000
_=/usr/local/bin/node-dev

NOTE: I removed a bunch of environment variables, but it listed a lot more

So you can also get the environment variables as well.

Anyway, I'm not super practiced on formatting the text of .sh commands to be perfect, but with the commands here you should provide everything needed.

the tr \\0 ' ' used for cmdline is for a space, which should be done on environtment variables as well, except i used \\n instead for display purposes.

[danschumann]

since what i just said doesn't give a perfect history of the command being ran, i'm currently reading http://serverfault.com/questions/526294/how-to-grab-either-bash-history-or-last-x-number-lines-from-shell-process-runnin

which is pretty in depth but i think claims to give the exact command from history

[danschumann]

that article is complex...

APID=25731; sudo gdb -batch --eval "attach $APID" --eval "call write_history(\"/tmp/bash_history-$APID.txt\")" --eval 'detach' --eval 'q'

that prints the history to that file in /tmp/bash_history-25731.txt , and the very last line is the EXACT command.

So, if you could figure out something better than call write_history, or just open that file and read the last line, that would be the exact last command.

I tried replacing call write_history with history, call history, etc, but it seems like the best way would be to open the file.

Also, i had to run gdb as super.. i don't know if this will still be the case, since potentially the uid of the bash pid would match tmux's, in which case you wouldn't have to sudo. This also requires installing gdb.

[danschumann]

regarding how to read the last line of that written history file, tail -1 /tmp/bash_history-25731.txt

so the full command to output only the last command in that pid is

APID=25731; sudo gdb -batch --eval "attach $APID" --eval "call write_history(\"/tmp/bash_history-$APID.txt\")" --eval 'detach' --eval 'q'; tail -1 "/tmp/bash_history-$APID.txt"

however, there is still a sudo in there and it requires installing gdb. So the previous method may be better, but it might work without sudo because the user running the command might match if tmux does it.

[danschumann]

writing '0' to /proc/sys/kernel/yama/ptrace_scope after installing gdb as the error suggests allows you to run the command without sudo

[danschumann]

pane_full_command() {
  pane_pid="$1"
  APID="$pane_pid"; gdb -batch --eval "attach $APID" --eval "call write_history(\"/tmp/bash_history-$APID.txt\")" --eval 'detach' --eval 'q' &>"/tmp/bb"; tail -1 "/tmp/bash_history-$APID.txt"   
  \tail -1 "/tmp/bash_history-$APID.txt"
}

is what i put in my /clone/path/scripts/save.sh, and it works! the thing is, I don't know if it only would recognize node in set -g @resurrect-processes

the &>"/tmp/bb" is just to suppress errors. there's probably a better way

[bruno-]

Hi @danschumann, this is awesome, thanks for all the investigation! gdb stuff blows my mind.

After reading everything you wrote, here are my thoughts:

• we have a couple different strategies for fetching the pane command. All have pros and cons.
• the proc filesystem strategy gives good results - but this one is available only on Linux (or at least it's not available on BSD systems, including OS X)
• the advanced strategy with 'gdb' seems to give the best results but requires installing gdb and some tweaking mentioned here.
• we might discover new & better ways how to do this in the future so it would be good to be prepared for this

Here are the suggestions how to move forward:

• instead of deciding to "fixate" on any particular way of doing things, let's enable multiple strategies with a sane default that "just works"
• with multiple options, we'll probably need to have a user option that will be honored. Having something like set -g @resurrect-save-command-strategy 'gdb' in tmux.conf. User specified strategy should always take precedence.
• we should probably set the default strategy to use proc filesystem. In case the procfs is not available, we can fall back to the current default using pgrep. Also consider the fallback to be the plain ps command as discussed here

@danschumann this shouldn't be too hard to implement. Give me a couple days and I'll ping you when it's done. Would you consider writing about the steps how to set gdb in a wiki? Something for the beginners.

[bruno-]

Hey @danschumann, strategies are now implemented (not documented yet though).

Here's how you can test drive the 'gdb' strategy:

• put this in 'tmux.conf' set -g @resurrect-save-command-strategy 'gdb'
• update the plugin (prefix + U if you're using TPM)
• when you now do resurrect save it should use the "gdb strategy" in this file.

Let me know how this works for you!

[danschumann]

I'll give this a shot when I get into work tomorrow! Exciting stuff On Sep 20, 2014 5:15 PM, "Bruno Sutic" notifications@github.com wrote:

Hey @danschumann https://github.com/danschumann, strategies are now implemented (not documented yet though).

Here's how you can test drive the 'gdb' strategy:

• put this in 'tmux.conf' set -g @resurrect-save-command-strategy 'gdb'
• update the plugin (prefix + U if you're using TPM)
• when you now do resurrect save it should use the "strategy" in this file https://github.com/tmux-plugins/tmux-resurrect/blob/master/save_command_strategies/gdb.sh#L14-15 .

Let me know how this works for you!

— Reply to this email directly or view it on GitHub https://github.com/tmux-plugins/tmux-resurrect/issues/44#issuecomment-56282128 .

[danschumann]

I started a wiki on my fork, but it doesn't look too easy to merge wiki's, though it is possible http://roman-ivanov.blogspot.com/2013/11/how-to-merge-github-wiki-changes-from.html

Otherwise, just enable wiki on here and I'll copy/paste the pages.

[danschumann]

here's a link to the wiki https://github.com/danschumann/tmux-resurrect/wiki

[bruno-]

Hi, I've enabled wikis for the repository. Feel free to copy the content whenever.

[danschumann]

Oddly enough now it's not working with the new code. Also either I changed my ptrace_scope back to a 1, or it automatically writes it.

0x00007fd24ac295cc in __libc_waitpid (pid=-1, stat_loc=0x7fff14da8ab8, options=10) at ../sysdeps/unix/sysv/linux/waitpid.c:31
31  ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
$1 = 0

That's what I get if I echo the result of the save_command_strategies/gdb#full_command gdb line

[bruno-]

Hmm, weird.. frankly I didn't do all the steps around enabling gdb on my machine. I just created a strategy based on your comment.

[bruno-]

Hey @danschumann , it's been a while since we've talked about this. It seems implementing the gdb didn't help resolve the issue. I'm now going to deprecate this functionality and schedule it for deletion in the next major release.

[unphased]

So I take it this crazy gdb approach only works with bash? Is it actually calling a "write_history" C method or something?

I would love it if I can resurrect my tmux running sudo nodemon app.js within zsh, but the whole pgrep -lf -P method just returns "sudo" (quite like how on linux tmux's process name for the pane is also "sudo", which freaking sucks)

[danschumann]

Yea, I had it working on mine, but when it came to the actual implementation it didn't work. Something got lost in the refactor cuz it was working.

[bruno-]

Hi guys, I'm not sure why the gdb strategy is not working.. I never had it set-up properly (I don't know how to work with gdb). Anyway, it seems this didn't peek the interest of a lot of people so I think it's best this feature stays scheduled for removal in the next major release.

[danschumann]

The only way I had it working was overriding things as I described early on in this issue ticket, and it was working, pasting the exact commands with all the arguments. After refactoring and creating strategies(and wiki), I can't confirm it was ever working. On Mar 14, 2015 9:55 AM, "Bruno Sutic" notifications@github.com wrote:

Hi guys, I'm not sure why the gdb strategy is not working.. I never had it set-up properly (I don't know how to work with gdb). Anyway, it seems this didn't peek the interest of a lot of people so I think it's best this feature stays scheduled for removal in the next major release.

— Reply to this email directly or view it on GitHub https://github.com/tmux-plugins/tmux-resurrect/issues/44#issuecomment-80496050 .

[unphased]

Hey guys I don't know if this is a really dense question or not, but I'm reading the readme and it has this section:

Start with tilde to restore a program whose process contains target name:

set -g @resurrect-processes 'irb pry "~rails server" "~rails console"'

Use -> to specify a command to be used when restoring a program (useful if the default restore command fails ):

set -g @resurrect-processes 'some_program "grunt->grunt development"'

And I'm kind of wondering (a) what is the additional class of situations that requires messing around with gdb and all that, and (b) what are the classes of situations that require the tilde and the arrow.

For example all I want right now is to implement restore rules for some web server stuff. What's the string that resurrect checks? I'm referring to what i should be putting on the left side of the ->. For example I know I can do something like tmux display-message -p '#T' and this will usually echo back the running command.

One example of a process i hope to resurrect is gulp. That one hopefully is a slam dunk!

But another one is far more complex, I usually run it in my zsh thus: sudo firewall-cmd --zone=public --add-port=8888/tcp && node flo-server.js.

Now I have tested tmux display-message -p '#T' on this and it just spits right back my entire command to me..... would I be able to add this to @resurrect-processes? Would I use the single quote method here? What then is the purpose of the arrow and the tilde modes?

Interestingly enough I tried "nodemon gulp node" in my @resurrect-processes and it succeeded with gulp and did something partially right with node (it ran node flo-server.js) so i can't really tell how the command is being reconstructed, whether it is something being queried out of tmux or not.

At any rate, I was able to get all 3 of my webserver processes persisted through a tmux reboot (hooray) by using some double-quoted entries and the one that has the && in it works fine if I just put this: "node flo-server.js->sudo firewall-cmd --zone=public --add-port=8888/tcp && node flo-server.js" so the semantics are actually pretty clear. Still wondering about the tilde though!

I found out that @bruno- already explained he uses tmux to get the pid and pgrep to get the running command. Simple enough. Thanks for everything! working great here on my end!

[bruno-]

Hey @unphased, lots of questions there.

uses tmux to get the pid and pgrep to get the running command

Actually, pid + ps is used by default. pgrep can be used with set -g @resurrect-save-command-strategy 'pgrep'

what is the additional class of situations that requires messing around with gdb and all that

Sometimes the pid + ps method does not succeed in detecting the exact command running inside tmux pane. We tried using gdb to detect those commands. It didn't really work. The gdb strategy is deprecated, so please don't use it.

what are the classes of situations that require the tilde and the arrow

Here's the example: the $ rails server command on my machine is detected as the following process /Users/bruno/.rbenv/versions/2.0.0-p481/bin/ruby script/rails server. This kind of "expansion" happens for a lot of processes..

/Users/bruno/.rbenv/versions/2.0.0-p481/bin/ruby script/rails server command will NOT be restored by default, so how do I instruct tmux to restore it? The config set -g @resurrect-processes 'rails server' will NOT work because detected process does not start with the rails server string.

The solution is to use tilde like this: set -g @resurrect-processes '"~rails server"'. That instructs tmux-resurrect to restore processes that have rails server string anywhere\ in saved process string (in /Users/bruno/.rbenv/versions/2.0.0-p481/bin/ruby script/rails server that string is at the end).

Now when I do a restore the command /Users/bruno/.rbenv/versions/2.0.0-p481/bin/ruby script/rails server will run in the pane. This is not really what I want.. I actually want just the rails server command to be executed.

To solution is to use this: set -g @resurrect-processes '"~rails server->rails server"'.

It seems you got the things running.. Let me know if you have more questions.

[unphased]

Awesome clear explanation, thanks. This should be in the readme!

[bruno-]

@unphased the explanation is added to this document