search

tmux-python / tmuxp

🖥️ Session manager for tmux, build on libtmux.
https://tmuxp.git-pull.com/
MIT License
4.04k stars 231 forks source link

tmux-resurrect/tmux-continuum alternative? #522

Open deliciouslytyped opened 4 years ago

deliciouslytyped commented 4 years ago

Hopefully this isn't off topic here; Does tmuxp provide an alternative to TPM/tmux-resurrect? I'd like something where all the programs aren't written in bash, and tmux-resurrect + tmux-continuum has some problems.

tmuxp's --freeze option sounds good, but at least restoring scrollback, and at least maybe for bash, somehow having per-pane history files would be nice. (I managed to get this working relatively ok for the above plugins once....)

Is tmuxp / libtmux appropriate for attempting a better reimplementation of these? Thanks!

deliciouslytyped commented 4 years ago

I'm a heavy, stateful user of the terminal and I have a lot of semi-finished things open in tmux a lot. It's really terrible when a reboot/battery depletion/crash makes me lose things. :( https://en.wikipedia.org/wiki/Persistence_(computer_science)#Orthogonal_or_transparent_persistence

tony commented 4 years ago

I haven't used tmux-resurrect. To my knowledge it also freezes processes, right?

tmuxp freeze works a bit differently. tmuxp doesn't preserve processes (though I guess we could add an ability to interoperate with tmux-resurrect?). tmuxp creates a new tmux session via commands to tmux and starts new shells.

If someone wants to PR optional support for that or anything in tmux-plugins, it's fine with me.

deliciouslytyped commented 4 years ago

No, to the best of my knowledge resurrect has an understandably rather naive restore model. I think it checks the running process via ps and then it uses some save/restore scripts to handle individual processes. I think there's some build in vim integration, but not arbitrary processes.

deliciouslytyped commented 4 years ago

My main priority would be restoring bash history and scrollback - the approach of resurrect should be possible to port for starters (assumption). I figured via adding appropriate functionality to the freeze sub-command.

deliciouslytyped commented 4 years ago

I'm working on restoring scrollback, I have dumping to a file minimally working. Would per-pane config file entries like this be appropriate, or do you have any better ideas?

"history": {    
            "shell_history" : file,
            "scrollback" : file
            }

My current idea is both of those file fields enable the history type for the pane if they are present, otherwise it's not used.

An alternative is to just have a global option for a dump directory.

The global dump directory option involves less file and editing noise - but this probably isn't relevant because the json file is basically autogenerated? The per-pane version is more flexible if someone wants. Both could also be done but then that needs a reasonably defined precedence semantics. (Global option unless stated otherwise?)

tmux-resurrect currently seems to have some thoughts about deprecating the shell history function due to it's "invasivity", would you accept the functionality?

I think the way it currently works for resurrect is that they autotype save/restore commands into the pane. I have some more ideas like setting the history file environment variables or attaching gdb to the bash process trees (similar to https://www.commandlinefu.com/commands/view/11427/dump-bash-command-history-of-an-active-shell-user.), but none of these seem like particularly good solutions, but I think the end result would still be nice even if it only works most of the time.

deliciouslytyped commented 4 years ago

Do you have some way we could discuss the design a bit? - and then I will summarize here for posterity.

tony commented 4 years ago

@deliciouslytyped

Up front - there are some aspects of this where tmux-resurrect overlaps with tmuxp kinda. On the tmux-resurrect readme, they mention migrating from tmuxinator (a similar thing to tmuxp). I have too much on my plate at the moment to go into resurrect in depth.

About chat: I got your email about that. Not enough to talk about yet, but in case it comes up in the future: I sent a mail to gitter.im asking for help getting orgs setup for tmux-python (and vcs-python). I tried setting it up a long time ago.

Regarding design:

I propose we find a way to add hook functionality to tmuxp.

We could use python import strings to allow arbitrary functions to execute python code, and we can pass libtmux objects inside.

plugins:
  - 'my_tmuxp_plugin.plugin.TmuxResurrect'
session_name: my session
windows:
- window_name: my window
  layout: tiled
  shell_command_before:
    - cd ~/                    # run as a first command in all panes
  panes:
    - echo pane
    - echo pane

Assume a git repo my_tmuxp_plugin with a directory my_tmuxp_plugin inside it, and a pypi package called my_tmuxp_plugin:

my_tmuxp_plugin/plugin.py:

class TmuxResurrect:
    def before_workspace_builder(self, session):
        # On blank workspace, after session created, but before any windows/panes/commands entered

        # technically, it'd be possible to do any kind of subprocess call here as well

    def before_script(self, session):
        # access to libtmux session object in tmuxp load

   def reattach(self, session):
       # if session_name already live, before reattaching

   def on_window_create(self, window):
        if window.window_name == 'my window':
            # do stuff

   def after_window_finished(self, window):  # after commands stuff commands finally entered
       pass

The thing with tmux resurrect though is 1.) they need to have it installed via TPM and/or have it in .tmux.conf.

Any ideas? Is my plugin idea a way you think tmux-resurrect / tmux plugins could be integrated into tmuxp?

tony commented 4 years ago

@deliciouslytyped I should also add I don't have much time to add / maintain features to tmuxp. I'm very lucky I had time to do some winter cleaning yesterday to CI *

I think the API thing makes sense since it'd allow linking into the libtmux under the hood and and pretty much doing anything. It'd also open up an area of community contribution and perhaps integration with other tmux plugins.

* I'm looking for maintainers actually https://github.com/tmux-python/tmuxp/issues/290 if you like working with python and tmux

deliciouslytyped commented 4 years ago

I'm pretty partial to IRC + a logging bot, but hey, it's your project. :D

I'm going to have to read over that again. I don't have any good ideas offhand.

My ability to work on things is also massively fluctuating and I'm not a huge fan of how my code usually ends up looking, so I'm not sure I could integrate nicely into the style of tmuxp. I kind of hoped I could get the core functionality working and then get some guidance on rough edges.

deliciouslytyped commented 4 years ago

This isn't necessariy an ideal time for me to rearchtect the core of tmuxp... :sweat_smile:

tony commented 4 years ago

@deliciouslytyped

This isn't necessariy an ideal time for me to rearchtect the core of tmuxp... 😅

I could stub out a plugin system to make libtmux objects available. Since I'm very familiar with the codebase, import strings, and libtmux. I could add tests and write a guide on making plugins.

What I can't address is writing tmux plugins themselves. Does that make sense?

deliciouslytyped commented 4 years ago

Sure.

tony commented 4 years ago

@deliciouslytyped The cool thing about this is if a plugin is added, it could/would be passed the Session, Window or Pane object being adjusted.

That is a very powerful API. Basically it means any python code could run, and they'd have access to libtmux.

I created an issue at #530 for a plugin system.

deliciouslytyped commented 4 years ago

For my current iteration of resurrect I just have this small snippet added to the inner pane loop of freeze() (using a patched verison of libtmux that has a more thorough capture_pane), and I haven't written any code for restoring yet.

           import os
            if "TMUXP_DUMP_PATH" in os.environ:
              dump_path = os.environ["TMUXP_DUMP_PATH"]
              dump_path = os.path.dirname( dump_path + "/" )
              try:
                os.mkdir(dump_path)
              except FileExistsError:
                pass
            if dump_path is not None:
              print("Dumping %s" % p)
              with open(os.path.join(dump_path, "%s-%s-%s" % (p["session_id"] , p["window_id"], p["pane_id"])), "wb") as f:
                val = p.capture_pane()
                for l in val:
                  f.write((l + "\n").encode("utf-8")) #TODO whats the correct way to handle the encoding? check some shell env var?
              print("Done dumping %s" % p)
tony commented 4 years ago

@deliciouslytyped Can you give me the git diff output of it?

Also, are there any other places inside of tmuxp you would find it helpful to "hook" into?

deliciouslytyped commented 4 years ago

Well, the problem is I don't know yet. The only things I can think of off the top of my head is that I'm going to need access to the above mentioned or similar config options, yeah? - or have the config options trigger my code? IDK - the usual possible variants on plugin infrastructure ...

So, both read access for restore, and write access for freeze.

I don't know what a good way to do this would be. A stack of pre-post hooks for config read-in and the config passed as a mutable variable (ugh) to every hook?

deliciouslytyped commented 4 years ago

I'll try to do some WIP PRs in a bit for libtmux and tmuxp, I guess.

tony commented 4 years ago

@deliciouslytyped Proposal: If you create an MVP and show a git diff (it's okay if its hacky), I can propose a way it'd be doable as a plugin #530

I'll try to do some WIP PRs in a bit for libtmux and tmuxp, I guess.

Yes, show a git diff for any changes to tmuxp/libtmux it takes to get tmux-resurrect working. It can be hard-coded. Also you can make a pull request for it and make it editable. Not sure if it's merge-able, but I can technically edit it directly then

deliciouslytyped commented 4 years ago

No promises but I'm going to be on a train for a few hours so with any luck I can throw something together. - modulo network access.

I'm not sure you can help with this much information, but this is just a simple restore, any idea about the problem?:

$ tmuxp load -S ./socket 3.json
[Loading] /run/user/1000/tmp.kSvepSlq6I/3.json
Traceback (most recent call last):
  File "/nix/store/hnnz9g0kswahg630b4rwxh00l8bw2pmd-tmuxp-1.5.4/bin/.tmuxp-wrapped", line 9, in <module>
    sys.exit(cli.cli())
  File "/nix/store/lma5qkl6r8vqrbhs7w202fxbrxd1c7fa-python3.7-click-7.0/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/lma5qkl6r8vqrbhs7w202fxbrxd1c7fa-python3.7-click-7.0/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/nix/store/lma5qkl6r8vqrbhs7w202fxbrxd1c7fa-python3.7-click-7.0/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/nix/store/lma5qkl6r8vqrbhs7w202fxbrxd1c7fa-python3.7-click-7.0/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/nix/store/lma5qkl6r8vqrbhs7w202fxbrxd1c7fa-python3.7-click-7.0/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/nix/store/lma5qkl6r8vqrbhs7w202fxbrxd1c7fa-python3.7-click-7.0/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/nix/store/hnnz9g0kswahg630b4rwxh00l8bw2pmd-tmuxp-1.5.4/lib/python3.7/site-packages/tmuxp/cli.py", line 814, in command_load
    load_workspace(config[-1], **tmux_options)
  File "/nix/store/hnnz9g0kswahg630b4rwxh00l8bw2pmd-tmuxp-1.5.4/lib/python3.7/site-packages/tmuxp/cli.py", line 536, in load_workspace
    builder.build()  # load tmux session via workspace builder
  File "/nix/store/hnnz9g0kswahg630b4rwxh00l8bw2pmd-tmuxp-1.5.4/lib/python3.7/site-packages/tmuxp/workspacebuilder.py", line 184, in build
    for p, pconf in self.iter_create_panes(w, wconf):
  File "/nix/store/hnnz9g0kswahg630b4rwxh00l8bw2pmd-tmuxp-1.5.4/lib/python3.7/site-packages/tmuxp/workspacebuilder.py", line 315, in iter_create_panes
    w.select_layout(wconf['layout'])
  File "/nix/store/khrmcyr697xxwi1f616q7w8frskbw1sx-python3.7-libtmux-0.8.2/lib/python3.7/site-packages/libtmux/window.py", line 151, in select_layout
    raise exc.LibTmuxException(proc.stderr)
libtmux.exc.LibTmuxException: ['server exited unexpectedly']

3.json:

{
  "session_name": "0",
  "windows": [
    {      
      "options": {},
      "window_name": "bash",
      "layout": "f72d,271x66,0,0[271x33,0,0,31,271x32,0,34,35]",
      "panes": [
        {
          "shell_command": "bash",
          "focus": "true"
        },
        "nix"
      ],
      "start_directory": "/bakery2/oven1/personal/projects/hask2/bb-clean"
    }
  ]  
}
deliciouslytyped commented 4 years ago

Sidenote: on the note of plugins, it might be interesting to create a CRIU plugin that can freeze certain processes, but it has a lot of limitations and I haven't figured out how to use it.

tony commented 4 years ago

@deliciouslytyped Does it still happen with a minimal config?

{
  "session_name": "0",
  "windows": [
    {      
      "options": {},
      "window_name": "name",
      "panes": [
        "bash"
      ]
    }
  ]  
}

the window isn't staying open

Does it work without -S?

Edit: I see this is related to layout: https://github.com/tmux-python/tmuxp/issues/531#issue-542363502

deliciouslytyped commented 4 years ago

I have a seemingly somewhat working codebase but the changes are a bit more extensive than I hoped, I'm going to be quite busy for a while so I might have to just dump a nasty diff / repo soon, so people can see it. I don't expect something to be done with it.

tony commented 4 years ago

@deliciouslytyped yeah a rough diff in a branch would be enough to look closer. If it's done in a PR, we don't have to merge, but the advantage is it runs through our CI. you can also allow maintainers to edit in the PR

deliciouslytyped commented 4 years ago

Sorry, this is really bad right now, but here it is, I just wanted to stop sitting on the code at least: archive.zip

There are some committed and uncommitted things, all on the main (checked out) branch.

tony commented 4 years ago

@deliciouslytyped Thank you! Can you create a fork of this repo and commit it to a branch? It's just what I'm used to since it's GH, easier to diff / make sense of. No problem if the code is rough - I just want a clear presentation of what's happening more than anything :)

Aqualie commented 4 years ago

Is this still being worked on by any chance? I'm currently looking for an alternative to tmux_continuum&resurrect as it's largely just a bunch of bash scripts that make alot of "noise" with my various security utilities..

deliciouslytyped commented 4 years ago

Still just the super messy stuff I have kludged up above - and I dont use it - it's not polished enough (not sure if it even works at all). I still have interest in this but haven't had a proper go at it since.

Are you interested in working on it?

Aqualie commented 4 years ago

I would be at present I don't have the time required to work on this it would be later this year before I could get around to being able to work on this.

unphased commented 4 years ago

Hi, I'm wary of adding more noise than contributing to the discussion, but this seems to be the right group of people to engage in this conversation with, so here goes.

I am a longtime user of resurrect (having abandoned continuum at some point a few years back for reasons I can't quite remember).

I am here because I have now decided that I'm a bit fed up with losing all the scrollback if I want to reboot. I'll list out my thoughts on the situation and what I would consider to be a step forward, keeping in mind the relative apparent difficulty of various hypothetical features. I'll also note I have very little exposure to tmuxp.

I think that's it. I don't want the holy grail, and I don't see anyone attempting a save/restore system for tmux that isn't some attempt at a holy grail, which proceeds well enough until it collapses under its own complexity.

All I need is a tool, whether it is integrated into tmux package manager or not, which has one command to do the following:

...and a command to do the following:

If I had this I'd have a lot less anxiety around rebooting. Having a restore system that is as staggeringly complex as it would need to be to do the "holy grail" restoration has very high likelihood of being brittle, and that would itself contribute to the anxiety.

It does seem that maybe itd be better to have this discussion in the resurrect repo as it already implements the first two bullets well enough for my purposes.

Edit: It looks like my rant is a bit moot due to https://github.com/tmux-plugins/tmux-resurrect/pull/79 (AND RTFM https://github.com/tmux-plugins/tmux-resurrect/blob/master/docs/restoring_pane_contents.md), sorry guys.

deliciouslytyped commented 4 years ago

I more or less agree with everything you said. The holy grail stuff was mostly me wondering out loud how good/bad it would work, and not too hard to try in isolation once surrounding tmuxp infrastructure exists (I think?). Indeed scrollback would get us a long way. Thank's for coherently stating a lot of my thoughts. And indeed, resurrect already covers a lot of this, but I had issues with it (can't remember either) and didn't want to start trying to complicate bash code further. :P A python implementation would also be a better base for further development to do whatever you want I think.

unphased commented 4 years ago

Yeah. @deliciouslytyped thanks for sharing your thoughts on my thoughts.

I'm gonna step out of the way now, but also leave a note that, now that I've gotten a clue and enabled the configuration parameter for saving pane contents, tmux-resurrect does indeed save it all into a gzip file under ~/.tmux/resurrect and I am fairly confident that restoring will work robustly, so it looks like I'll be sticking with resurrect. Maybe the next step for me would be to venture back into continuum territory to try to save periodically but my buffer size is so large and my panes so numerous that it will probably be too expensive to be practical to persist periodically.

deliciouslytyped commented 4 years ago

Also I haven't looked in a while but tmuxp seemed a lot more maintained than the entire tmux plugin manager ecosystem, but I might have misinterpreted. Which is not to say that it's a good idea to try to dump everything into tmuxp, but...something.

deliciouslytyped commented 4 years ago

I think I've had resurrect corrupt it's storage and fail to restore before.

joseph-flinn commented 3 years ago

As of tmuxp 1.7.0, we have a plugin system based off of what tony designed above. This might (or might not) be helpful in any continued work on a more stateful session freeze.

deliciouslytyped commented 3 years ago

Hey, cool! I'll try to give it a look at the beginning of February. Hopefully it does what I want :)

deliciouslytyped commented 3 years ago

@joseph-flinn I haven't looked deep enough yet, and I need to look through what I was talking about a year ago. Untii then, do you have any suggestions for modifying the functionality of freeze and load? I think there are 3 main components needed to implement resurrect:

joseph-flinn commented 3 years ago

@deliciouslytyped I would actually caution against modifying the current functionality of the freeze and load commands. They are very specific to freezing and loading the current layouts and starting points for each profile.

I would instead look at the Plugin System API to write a plugin that will save your current session in a file on exit and load from that file; similar to an automatic vim mksession. All of the functionality is not currently in the Plugin System to hook into the "on_exit" yet, but there is a conversation going on adding that over in https://github.com/tmux-python/tmuxp/issues/516. It is one of the things that I would like to tackle next.

Going the plugin route will save current functionality for the users who are not interested in saving tty session state or shell history state. This is how I would approach it.

deliciouslytyped commented 3 years ago

Yeah, if I was less tired I'd be a bit more coherent. :) I did mean to say that I would like to use the plugin system if possible, to implement functionality equivalent to such modifications. Thanks.

Note to self: when the basic functionality is working, look into an "append" mode for scrollback.

joseph-flinn commented 3 years ago

Ah, I apologize for the misunderstanding.

Thought dump

With the current functionality of the plugin system, I would approach restoring the pane sessions on a window by window basis using https://tmuxp.git-pull.com/plugin_system.html#tmuxp.plugin.TmuxpPlugin.after_window_finished. In a perfect world, we would add in another plugin hook after_pane_finished which would allow more granularity to make life easier. And we would need to write a hook for the plugin system for when the session is exited/killed.

My gut feeling is that to build the resurrect session file, there is going to be a lot of digging around the internal of tmux itself to figure out how to identify the correct pane to load the resurrect session object (at least, this would be the case for me).

How I would approach building the plugin

mdeguzis commented 2 years ago

This project already saves the base command you are in right? So if you can capture the PID, just pull it out of /prod/$PID/cmdline? This would at least give me the current things I was working on for editors and such so I don't lose my place. Breaking up the goals here into milestones and separate issues would be ideal, as there are a few different asks here.

I might hack my copy to at least add this if I can, but not sure how much I need to dig into things yet.

Null bytes are written, so you just need to convert them:

$ sed 's/\x0/\ /g' /proc/13392/cmdline 
vim /tmp/a
$ cat ~/.tmuxp/fun.yaml 
session_name: fun
windows:
- focus: 'true'
  layout: 99ae,146x37,0,0[146x14,0,0,0,146x18,0,15,2,146x3,0,34,1]
  options: {}
  panes:
  - focus: 'true'
    shell_command:
    - cd /tmp
    - bash
  - shell_command:
    - cd /home/mikeyd
    - bash
  - shell_command:
    - cd /home/mikeyd
    - vim

I'm guessing this project pulls the base command to save with:

[mikeyd@pop-os: ~]$ tmux list-panes -t temp -F '#{pane_current_command}'
tmux
vim
tony commented 2 years ago

I'm still open to a PR on this, also I'm open to tmuxp becoming a tpm plugin, etc.

All options are open if we have champion who can PR it, so long as there's nice tests and documentation.

mdeguzis commented 2 years ago

I'll play around with this in my spare time. No promises or sole ownership at the time being, as I think any good idea for this is worth pursuing. I think grabbing the PID of the pane somehow and using /proc is worth pursuing.

mdeguzis commented 2 years ago

I've been looking at libtmux and panes.py. I saw formats.py has what seems to be the exposed tmux variables, but I can't seem to print p.pane_pid in tmuxp. How are the variables in libtmux exposed? Are there dev docs I can read? Maybe I missed them.