openvstorage / volumedriver

The Open vStorage VolumeDriver is the core of the Open vStorage solution: a high performance distributed block layer. It converts block storage into objects (Storage Container Objects).
Other
37 stars 18 forks source link

[Python client] restart//stop hang while info_volume returned #393

Closed JeffreyDevloo closed 5 years ago

JeffreyDevloo commented 5 years ago

Problem description

Both the StorageRouterClient and LocalStorageRouterClient were hanging when using either restart_volume or stop_object without specifying a timeout. After keyboard interrupting using ctrl-c, nothing happened. I also checked info_volume which yielded me a return value

Tailing the logging in the volumedriver itself: the command were received and processed. It looks like the python client was stuck somewhere.

Restarting the volumedriver while the python cleint was stuck raised the KeyboardInterrupt.

redlicha commented 5 years ago

The underlying issue is most likely that the Python client was waiting in a C++ call for a response from the volumedriver (without timeout). While in a C(++) call, Python signal handlers are not run, only after the call. Cf. https://docs.python.org/3.6/library/signal.html#execution-of-python-signal-handlers (Python 3):

A Python signal handler does not get executed inside the low-level (C) signal handler. Instead, the low-level signal handler sets a flag which tells the virtual machine to execute the corresponding Python signal handler at a later point(for example at the next bytecode instruction). This has consequences:

It makes little sense to catch synchronous errors like SIGFPE or SIGSEGV that are caused by an invalid operation in C code. Python will return from the signal handler to the C code, which is likely to raise the same signal again, causing Python to apparently hang. From Python 3.3 onwards, you can use the faulthandler module to report on synchronous errors. A long-running calculation implemented purely in C (such as regular expression matching on a large body of text) may run uninterrupted for an arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes.

JeffreyDevloo commented 5 years ago

This issue was moved to openvstorage/volumedriver-ee#165