BUG: Temserver freeze

[kferjaoui]

I happens that the server_tem.py console freezes. I do not think that this is due to bad error handling on the gui side... is the command msgs = self.socket.recv_multipart() blocking? anything to do with the sequential nature of the REP socket? may be running the TEMServer::_run() method in a separate thread would avoid freezes?

[erikfrojdh]

What do you need to do to unblock the server? Just hit tab or do you need to kill it?

[kferjaoui]

What do you need to do to unblock the server? Just hit tab or do you need to kill it?

Just hitting tab awakes the server hence unblocking the GUI too..

[erikfrojdh]

Ok will investigate. I haven't seen this on linux, but then again it's not running with the real PyJEM.

[erikfrojdh]

Could be related to quick edit mode in the console?

• https://stackoverflow.com/questions/63254323/python-loop-keeps-stopping-in-windows-interpreter
• https://stackoverflow.com/questions/37500076/how-to-enable-windows-console-quickedit-mode-from-python

[kferjaoui]

Steps to Disable Quick Edit Mode:

1.Open the Command Prompt where you are running your script.
2.Right-click on the title bar of the Command Prompt window.
3.Select Properties from the dropdown menu.
4.In the Options tab, uncheck the box for Quick Edit Mode.
5.Click OK to apply the changes.

[erikfrojdh]

import ctypes

kernel32 = ctypes.windll.kernel32
kernel32.SetConsoleMode(kernel32.GetStdHandle(-10), 128)

This might as well work but I haven't tested it.

@kferjaoui and @KiyofumiTakaba if you get to test any of the methods it would be great. If it doesn't work then we need to keep looking for the issue

[kferjaoui]

So far no issues after disabling the Quick edit... will keep testing

[kferjaoui]

So far no issues after disabling the Quick edit... will keep testing

I had the temserver console freeze for may be 2 secs, throwing in the meantime a timeout error handled by the GUI, then the server_tem awakes by itself with the command being processed despite the error raised (ref. the logging below).

On the GUI side:

2024-10-01 15:50:05 - REQ: b'SetXRel', b'[-10000]'
2024-10-01 15:50:05 - REP: OK:None
2024-10-01 15:50:08 - REQ: b'SetTXRel', b'[10]'
2024-10-01 15:50:08 - REP: OK:None
2024-10-01 15:50:10 - REQ: b'SetXRel', b'[10000]'
2024-10-01 15:50:14 - REP: OK:None
2024-10-01 15:50:14 - REQ: b'SetTXRel', b'[-10]'
Traceback (most recent call last):
  File "/home/psi/erik/simple-tem/simple_tem/TEMClient.py", line 36, in _send_message
    reply = socket.recv_multipart()
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psi/miniconda3/envs/v8/lib/python3.11/site-packages/zmq/sugar/socket.py", line 805, in recv_multipart
    parts = [self.recv(flags, copy=copy, track=track)]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "zmq/backend/cython/socket.pyx", line 805, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 841, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 199, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/socket.pyx", line 194, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/checkrc.pxd", line 22, in zmq.backend.cython.checkrc._check_rc
zmq.error.Again: Resource temporarily unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/psi/software/viewer_2.0/GUI/jungfrau_gui/ui_components/tem_controls/tem_action.py", line 65, in <lambda>
    lambda: self.control.client.SetTXRel(-10))        
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psi/erik/simple-tem/simple_tem/TEMClient.py", line 185, in SetTXRel
    self._send_message("SetTXRel", val)
  File "/home/psi/erik/simple-tem/simple_tem/TEMClient.py", line 43, in _send_message
    raise TimeoutError(f"Timeout while waiting for reply from {self.host}:{self.port}")
TimeoutError: Timeout while waiting for reply from temserver:3535
2024-10-01 15:50:34 - REQ: b'SetTXRel', b'[-10]'
2024-10-01 15:50:34 - REP: OK:None
    2024-10-01 15:50:51 - REQ: b'SetTXRel', b'[10]'
2024-10-01 15:50:51 - REP: OK:None

while on the server_tem.py (windows powershell) side:

2024-10-01 15:48:50 - REQ: SetXRel, [-10000]
2024-10-01 15:48:50 - REP: b'"OK"', b'null'
2024-10-01 15:48:52 - REQ: SetTXRel, [10]
2024-10-01 15:48:52 - REP: b'"OK"', b'null'
2024-10-01 15:48:55 - REQ: SetXRel, [10000]
2024-10-01 15:48:58 - REP: b'"OK"', b'null'
2024-10-01 15:48:58 - REQ: SetTXRel, [-10]
2024-10-01 15:49:04 - REP: b'"OK"', b'null'
2024-10-01 15:49:19 - REQ: SetTXRel, [-10]
2024-10-01 15:49:19 - REP: b'"OK"', b'null'
2024-10-01 15:49:36 - REQ: SetTXRel, [10]
2024-10-01 15:49:36 - REP: b'"OK"', b'null'

[erikfrojdh]

Did the tem server recover by itself? Then we could increase the timeout. Will make the GUI unresponsive but not for so long. Or run commands get run in a separate thread...

[kferjaoui]

Yes the command prompt recovered by itself (Quick Edit disabled so couldn't press tab...)

One case scenario, where increasing the timeout would be very helpful, is typically when the user clicks on the button '+10deg' (at '2deg/s' for e.g.) and clicks again on it before the first rotation ends, a timeout error is thrown... If I understand correctly this is due to the REQ-REP behavior of the socket right?

A work around the freeze of the timeout would be if we defined a method that takes a tem command and make the call concurrently:

def thread_tem_method(tem_command, *args):
    client = TEMClient("temserver", 3535)
    method = getattr(client, tem_command)
    rep = method(*args)
    return rep

threading.Thread(target=thread_tem_method, args=(method_name, *args), daemon=True).start()

can do better for the impleentation but what do you think?

[erikfrojdh]

The call should return before the rotation ends to I don't think this is an issue.

[kferjaoui]

The call should return before the rotation ends to I don't think this is an issue.

This does not seem to be true for SetTXRel()

[erikfrojdh]

sole freeze for may be 2 secs, throwing in the meantime a timeout error handled by the GUI, then the server_tem awakes by itself with the command being processed despite the error raised (ref. the logging below).

This is something we need to test. For the absolute rotation my impression is that the call returns immediately, or at least way before the rotation is done.

[kferjaoui]

Yes for the absolute rotation no delays or freezes. Another option, i guess, is to use the SetTiltXAngle for the relative rotations

The call should return before the rotation ends to I don't think this is an issue.

This does not seem to be true for SetTXRel()

On top of the freezes, I also saw that, sometimes, clicking right after SetTXRel() has been sent, causes the rotation to interrupt