search

OpenSIPS / opensips-cli

OpenSIPS CLI tool - an interactive command line tool that can be used to control and monitor OpenSIPS servers.
GNU General Public License v3.0
85 stars 48 forks source link

CLI crashes on mi method subs_phtable_list #72

Open staskobzar opened 3 years ago

staskobzar commented 3 years ago

Hello,

Utility crash on comman "mi subs_phtable_list":

(opensips-cli@http): mi subs_phtable_list
Traceback (most recent call last):
  File "/usr/lib/python3.8/cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)

Here is full crash output: https://www.dropbox.com/s/w2ythwwy9ozn398/crash-opensips-cli.log?dl=0

OS: Ubuntu 20.04 focal Python 3.8.5 OpenSIPS CLI 0.1.0 installed from deb packages: https://apt.opensips.org/packages.php?v=cli

I am using http mi connection. Here is my configuration file:

[default]
log_level: WARNING
prompt_name: opensips-cli@http
prompt_intro: OpenSIPS MI CLI
prompt_emptyline_repeat_cmd: False
history_file: ~/.opensips-cli.history
history_file_size: 10000
output_type: lines
communication_type: http
url: http://127.0.0.1:8000/mi

Here is network traffic fetched when command issued: https://www.dropbox.com/s/h3u238814xs0zwg/crash-opensips-cli-net.log?dl=0

Thank you! Stas

bogdan-iancu commented 3 years ago

Hi @staskobzar , could you take the latest CLI (use the nightly builds from repo) and re-confirm this crash?

staskobzar commented 3 years ago

Hi @bogdan-iancu , I tried with nightly build:

Get:9 https://apt.opensips.org focal/cli-nightly amd64 opensips-cli all 0.1~20201228~e366d33-1 [36.1 kB]

And I have the same problem, unfortunatly:

[ 18:33:33 ] ❯ opensips-cli
ClusterPBX OpenSIPS MI CLI
(opensips-cli@http): mi uptime
Now: Fri Jan  8 18:33:47 2021
Up since: Fri Jan  8 18:23:00 2021
Up time: 647 [sec]
(opensips-cli@http): mi subs_phtable_list
Traceback (most recent call last):
  File "/usr/lib/python3.8/cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)
AttributeError: 'OpenSIPSCLIShell' object has no attribute 'do_mi'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/opensips-cli", line 9, in <module>
    run_console()
  File "/usr/bin/opensips-cli", line 6, in run_console
    main.main()
  File "/usr/local/lib/python3.8/dist-packages/opensipscli-0.1.0-py3.8.egg/opensipscli/main.py", line 78, in main
    sys.exit(shell.cmdloop())
  File "/usr/local/lib/python3.8/dist-packages/opensipscli-0.1.0-py3.8.egg/opensipscli/cli.py", line 253, in cmdloop
    super(OpenSIPSCLIShell, self).cmdloop(intro='')
  File "/usr/lib/python3.8/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/usr/lib/python3.8/cmd.py", line 216, in onecmd
    return self.default(line)
  File "/usr/local/lib/python3.8/dist-packages/opensipscli-0.1.0-py3.8.egg/opensipscli/cli.py", line 374, in default
    self.run_command(module, cmd, params)
  File "/usr/local/lib/python3.8/dist-packages/opensipscli-0.1.0-py3.8.egg/opensipscli/cli.py", line 357, in run_command
    return mod[0].__invoke__(cmd, params)
  File "/usr/local/lib/python3.8/dist-packages/opensipscli-0.1.0-py3.8.egg/opensipscli/modules/mi.py", line 131, in __invoke__
    res = comm.execute(cmd, params)
  File "/usr/local/lib/python3.8/dist-packages/opensipscli-0.1.0-py3.8.egg/opensipscli/comm.py", line 40, in execute
    ret = comm_handler.execute(cmd, params)
  File "/usr/local/lib/python3.8/dist-packages/opensipscli-0.1.0-py3.8.egg/opensipscli/communication/http.py", line 38, in execute
    replycmd = urllib.request.urlopen(request).read().decode()
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
     return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1379, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

I have tried with source from git repo and the same error.

bogdan-iancu commented 3 years ago

@staskobzar, if I correctly read that backtrace, the relevant part is "Remote end closed connection without response" - could you confirm this by doing a tcpdump on the httpd port of OpenSIPS (for JSONRPC) ? Maybe there is an error in OpenSIPS, preventing any reply back to CLI.

staskobzar commented 3 years ago

@bogdan-iancu yes, you are right. I have opensips crash this in logs:

Jan 12 12:39:49 mgm opensips[1978]: CRITICAL:core:sig_usr: segfault in process pid: 1978, id: 3
Jan 12 12:39:50 mgm opensips[1975]: INFO:core:handle_sigs: child process 1978 exited by a signal 11
Jan 12 12:39:50 mgm opensips[1975]: INFO:core:handle_sigs: core was generated
Jan 12 12:39:50 mgm opensips[1975]: INFO:core:handle_sigs: terminating due to SIGCHLD
Jan 12 12:39:50 mgm opensips[1976]: INFO:core:sig_usr: signal 15 received

Here is a link to full backtrack output from core dump: https://www.dropbox.com/s/898om91s55vn43r/core-2021-01-12-opensips-cli-crash-issue-72.txt?dl=0

staskobzar commented 3 years ago

@bogdan-iancu it might be connected to the issue I have reported for opensips: https://github.com/OpenSIPS/opensips/issues/2267

When I enable presence in opensips I have it crashed. And it looks like it also reports on lock. I am not sure but still... Here the core dump I made today: https://www.dropbox.com/s/ubtbpd47ncu6shk/core-issue-2267-bt-full-2021-01-12.txt?dl=0

version: opensips 3.1.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 3f58031ef
main.c compiled on 11:50:08 Jan  8 2021 with gcc 9
bogdan-iancu commented 3 years ago

@staskobzar , yes, this is indeed primarily an OpenSIPS issue, so let's move back to the #2267 opensips ticket. Nevertheless, the CLI should be able to better handle such situations (dropping JSONRPC conns) without the nasty backtraces. Maybe one of the developers of CLI can look into, @razvancrainea , @liviuchircu

staskobzar commented 3 years ago

@bogdan-iancu got it. Thank you!