`TmuxObjectDoesNotExist` when creating `libtmux.Server().new_session(attach=True)`

[Pipeliner]

$ python3 -c 'import libtmux ; print(libtmux.Server().new_session(attach=True))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vadim/.cache/pypoetry/virtualenvs/robots-ml-agent-PwTI9jHQ-py3.8/lib/python3.8/site-packages/libtmux/server.py", line 493, in new_session
    return Session.from_session_id(
  File "/home/vadim/.cache/pypoetry/virtualenvs/robots-ml-agent-PwTI9jHQ-py3.8/lib/python3.8/site-packages/libtmux/session.py", line 85, in from_session_id
    session = fetch_obj(
  File "/home/vadim/.cache/pypoetry/virtualenvs/robots-ml-agent-PwTI9jHQ-py3.8/lib/python3.8/site-packages/libtmux/neo.py", line 240, in fetch_obj
    raise exc.TmuxObjectDoesNotExist(
libtmux.exc.TmuxObjectDoesNotExist: Could not find object

from pyproject.toml:
[tool.poetry.dependencies]
python = "^3.8"
libtmux = {git = "https://github.com/tmux-python/libtmux/"}
tmuxp = "^1.31.0"

$ tmux -V
tmux 3.0a

$ tmuxp -V
tmuxp 1.31.0, libtmux 0.23.2

$ apt-cache policy tmux
tmux:
  Installed: 3.0a-2ubuntu0.4
  Candidate: 3.0a-2ubuntu0.4
  Version table:
 *** 3.0a-2ubuntu0.4 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status
     3.0a-2 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=en_US.utf8

When I run this code, the session is created successfully (tmux opens up, there is a single window with text like $2 in it, closed upon enter, and then I have a shell) but libtmux does not seem to find it.

It's the same with tmux 3.3a from Linuxbrew. When the session gets started:

tmux list-sessions
0: 1 windows (created Sat Oct 21 02:13:23 2023) (attached)
2: 1 windows (created Sat Oct 21 02:13:48 2023) (attached)

[Pipeliner]

With older version of libtmux, the error is different:

python3 -c 'import libtmux ; print(libtmux.Server().new_session(attach=True, session_name="r"))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/libtmux/server.py", line 567, in new_session
    session = Session(server=self, **session)
  File "/usr/lib/python3/dist-packages/libtmux/session.py", line 57, in __init__
    raise ValueError('Session requires a `session_id`')
ValueError: Session requires a `session_id`

[tony]

@Pipeliner I can't diagnose remotely, but a decision tree I'd consider:

• Do you have a tmux config file? (if your environment did, it would be implicitly used)
• Does libtmux.Server(config_file='/dev/null)` change anything?
• Does libtmux.Server().is_alive() return anything?

From libtmux's side: I think it'd be nice if libtmux offered more useful debug output in cases like this.

[tony]

When I try this:

python3 -c 'import libtmux ; print(libtmux.Server().new_session(attach=True, session_name="r", socket_name="test_test"))'

When I this my session persists and I can use tmux normally.

When I run this code, the session is created successfully (tmux opens up, there is a single window with text like $2 in it, closed upon enter, and then I have a shell) but libtmux does not seem to find it.

Can you restate this? e.g. is the tmux client (and ostensibly the newly created session) automatically killed after entering? Or are you detaching the client?

[Pipeliner]

Do you have a tmux config file? (if your environment did, it would be implicitly used)

I'll have to check, I don't remember configuring it.

Does libtmux.Server(config_file='/dev/null)` change anything?

No, same behavior

Does libtmux.Server().is_alive() return anything?

True

When I run this code, the session is created successfully (tmux opens up, there is a single window with text like $2 in it, closed upon enter, and then I have a shell) but libtmux does not seem to find it.

Restated:

1. New tmux session is created
2. It has a single window
3. The window has text like $2
4. When I press enter, the text disappears, and I have a shell inside this tmux new session window.
5. When I close the window with ^D, I get the error I mentioned: TmuxObjectDoesNotExist

[tony]

Thank you for the above!

When I run this code, the session is created successfully (tmux opens up, there is a single window with text like $2 in it, closed upon enter, and then I have a shell) but libtmux does not seem to find it.

Restated:

1. New tmux session is created
2. It has a single window
3. The window has text like $2
4. When I press enter, the text disappears, and I have a shell inside this tmux new session window.
5. When I close the window with ^D, I get the error I mentioned: TmuxObjectDoesNotExist

Recreated.

python3 -c 'import libtmux ; print(libtmux.Server().new_session(attach=True, session_name="r"))'

@Pipeliner If I were to guess, it may be that the python script itself is spawning tmux(1), so if you were to use htop or top:

Here is what I get:

  35301 pts/10   Ss     0:00 -zsh
  37453 pts/10   S+     0:00  \_ python3 -c import libtmux ; print(libtmux.Server().new_session(attach=True, session_name="r", socket_name="test_test", config_file="/dev/null"))
  37559 pts/10   S+     0:00      \_ /usr/bin/tmux new-session -P -F#{session_id} -sr

So it looks like libtmux is trying to populate its attributes by returning a Session object, but by the time that code runs, 37559 is closed.

Interesting. I'd need to think about the preferred way to handle this. Usually, we're focused on returning the python object (the libtmux session), but the session itself is destroyed before it can even return.

[tony]

With a fresh tmux server (socket_name="test_test") and vanilla configuration (config_file="/dev/null"):

python3 -c 'import libtmux ; print(libtmux.Server(socket_name="test_test", config_file="/dev/null").new_session(attach=True, session_name="r"))'

Closing will show this (same idea, though):

❯ python3 -c 'import libtmux ; print(libtmux.Server(socket_name="test_test", config_file="/dev/null").new_session(attach=True, session_name="r"))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".local/lib/python3.12/site-packages/libtmux/server.py", line 493, in new_session
    return Session.from_session_id(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".local/lib/python3.12/site-packages/libtmux/session.py", line 85, in from_session_id
    session = fetch_obj(
              ^^^^^^^^^^
  File .local/lib/python3.12/site-packages/libtmux/neo.py", line 232, in fetch_obj
    obj_formatters_filtered = fetch_objs(
                              ^^^^^^^^^^^
  File ".local/lib/python3.12/site-packages/libtmux/neo.py", line 208, in fetch_objs
    raise exc.LibTmuxException(proc.stderr)
libtmux.exc.LibTmuxException: ['no server running on /tmp/tmux-1000/test_test']

[tony]

@Pipeliner Try this:

python3 -c 'import libtmux ; session = libtmux.Server().new_session(session_name="r"); print(session); session.attach_session()'

Explanation of above: Return the new_session, print the object, then attach. This way its not trying to print the same entity its' return statement destroyed.

Same, but with independent server + vanilla config file:

python3 -c 'import libtmux ; session = libtmux.Server(socket_name="test_test", config_file="/dev/null").new_session(session_name="r"); print(session); session.attach_session()'

[Pipeliner]

Your samples works @tony thank you! So it seems that attach=True does not work for new_session()?

I am stuck trying to make further progress though. I can't either create a window or access an existing one.

Sometimes I get this exception though it's probably not directly related to my problems:

--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit
    msg = self.format(record)
    File "/usr/lib/python3.8/logging/__init__.py", line 929, in format
      return fmt.format(record)
      File "/usr/lib/python3.8/logging/__init__.py", line 668, in format
       record.message = record.getMessage()
       File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage
        msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "main.py", line 66, in <module>
    agent = RobotGameAgent()
  File "main.py", line 15, in __init__
    self.session = self.server.new_session(
  File "/home/vadim/.cache/pypoetry/virtualenvs/robots-ml-agent-PwTI9jHQ-py3.8/lib/python3.8/site-packages/libtmux/server.py", line 481, in new_session
    logger.error("new-session", *tmux_args)
Message: 'new-session'
Arguments: ('-P', '-F#{session_id}', '-sRobotsGameSession', '-d', '-n', 'RobotGameWindow', '/usr/game/robots')

The script I am trying to make run (sorry for lots of commented code, I was just trying to make it work using any means possible):

#!/usr/bin/env python3
import os
import time
import random
import libtmux

class RobotGameAgent:
    def __init__(self):
        self.moves = ["y", "k", "u", "h", "l", "b", "j", "n", "w", "t"]
        self.server = libtmux.Server()
        # self.server.attach_session("0")
        print(self.server.sessions)
        # self.session = self.server.sessions[0]
        self.session = self.server.new_session(
            session_name="RobotsGameSession",
            # attach=True,
            kill_session=True,
            # session_id="RobotGameSession",
            window_name="RobotGameWindow",
            window_command="/usr/game/robots",
        )
        self.session.attach_session()
        # self.window = self.session.new_window(
        #     # attach=True,
        #     window_name="RobotsGameWindow",
        #     window_shell="/usr/game/robots",
        # )

        self.window = self.session.attached_window
        self.pane = self.window.attached_pane
        # self.pane.pane_start_command("/usr/game/robots")
        self.pane.send_keys("robots\n")

    def get_game_state(self):
        # Give the game some time to process
        time.sleep(5)

        # Capture the pane content
        pane_content = self.pane.capture_pane()

        # The game state is now stored in `pane_content`
        return pane_content

    def choose_move(self, game_state):
        # This is where you would add your game-playing logic
        # For now, we'll just choose a move randomly
        return random.choice(self.moves)

    def make_move(self, move):
        # Send the move to the game
        self.pane.send_keys(move)

    def play_turn(self):
        # Get the current game state
        game_state = self.get_game_state()

        # Choose a move based on the current game state
        move = self.choose_move(game_state)

        # Make the chosen move
        self.make_move(move)

# Create an instance of the agent and have it play a turn
agent = RobotGameAgent()
for _ in range(5):
    agent.play_turn()
    print("\n".join(agent.get_game_state()))
input("Press enter to exit")

No matter what I do, I get either libtmux.exc.LibTmuxException: ["can't find session: $24"] or libtmux.exc.TmuxObjectDoesNotExist: Could not find object

[tony]

Your samples works @tony thank you! So it seems that attach=True does not work for new_session()?

As of v0.23.2: new_session() with attach=True will start a tmux client inside the python script itself. That means if nothing else is keeping the session alive, that error will show.

Can you think of a behavior that'd be more intuitive than that?

I will continue with the rest of the answer next (though it may need to be next weekend as I get sleepy around this time)

[tony]

@Pipeliner Added formatting to your comment here

[Pipeliner]

That means if nothing else is keeping the session alive, that error will show. Can you think of a behavior that'd be more intuitive than that?

It seems I'm generally confused by what's happening. I don't get it why the session would disappear right after creation and before new_session() has a chance to find it. Anyways, if this is an expected behavior, an exception with more specific message like "the session you created is already gone because you created it the wrong way" is more helpful than generic libtmux.exc.LibTmuxException: ["can't find session: $24"] or libtmux.exc.TmuxObjectDoesNotExist: Could not find object.

Thanks for your help so much!

[tony]

Something needs to be done, along the lines of a os.fork / os.spawn / etc.

The solution to this would actually make a good example for the documentation.

Below, all I did was move attach_session() down and add a best_window/best_pane property. For it to work, it needs to be decided when libtmux will actually be running. My guess is you may want to do something to fork self.attach_session() so it runs separately and doesn't block the script.

#!/usr/bin/env python3
import os
import time
import random
import libtmux

class RobotGameAgent:
    def __init__(self):
        self.moves = ["y", "k", "u", "h", "l", "b", "j", "n", "w", "t"]
        self.server = libtmux.Server()
        # self.server.attach_session("0")
        print(self.server.sessions)
        # self.session = self.server.sessions[0]
        self.session = self.server.new_session(
            session_name="RobotsGameSession",
            # attach=True,
            kill_session=True,
            # session_id="RobotGameSession",
            window_name="RobotGameWindow",
            window_command="/usr/game/robots",
        )
        # self.window = self.session.new_window(
        #     # attach=True,
        #     window_name="RobotsGameWindow",
        #     window_shell="/usr/game/robots",
        # )

        self.window = self.best_window
        self.pane = self.best_pane
        # self.pane.pane_start_command("/usr/game/robots")
        self.pane.send_keys("robots\n")

        self.session.attach_session()

    @property
    def best_window(self):
        try:
            return self.session.attached_window
        except libtmux.exc.LibTmuxException:
            return self.session.windows[0]

    @property
    def best_pane(self):
        window = self.best_window
        try:
            return window.attached_pane
        except libtmux.exc.LibTmuxException:
            return window.panes[0]

    def get_game_state(self):
        # Give the game some time to process
        time.sleep(5)

        # Capture the pane content
        pane_content = self.pane.capture_pane()

        # The game state is now stored in `pane_content`
        return pane_content

    def choose_move(self, game_state):
        # This is where you would add your game-playing logic
        # For now, we'll just choose a move randomly
        return random.choice(self.moves)

    def make_move(self, move):
        # Send the move to the game
        self.pane.send_keys(move)

    def play_turn(self):
        # Get the current game state
        game_state = self.get_game_state()

        # Choose a move based on the current game state
        move = self.choose_move(game_state)

        # Make the chosen move
        self.make_move(move)

# Create an instance of the agent and have it play a turn
agent = RobotGameAgent()
for _ in range(5):
    agent.play_turn()
    print("\n".join(agent.get_game_state()))
input("Press enter to exit")

[Pipeliner]

Thank you, your version works. I had to remove self.session.attach_session() though because it blocks the script (this is unexpected and as far as I can see is not clearly documented). I have tried calling this from os.fork but it leads to some glitches with handling keyboard input. os.spawn may be a better fit, I might try it later.

My confusion about .new_session(attach=True) is twofold:

1. Is there a situation when it works, and how is the way I was calling it "wrong" i.e. different from that scenario?
2. If I am using it the "wrong" way, how should I figure that from the error message / exception I get?