OpenSIPS CLI not printing binary output

[MayamaTakeshi]

Commit ed710d5cf0c1be9676c3c2b8926f2e40756e1915

$ opensips-cli -x mi ul_dump SQLAlchemy version: 1.3.3 Traceback (most recent call last): File "/usr/local/bin/opensips-cli", line 4, in import('pkg_resources').run_script('opensipscli==0.1', 'opensips-cli') File "/usr/lib/python3/dist-packages/pkg_resources.py", line 534, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/lib/python3/dist-packages/pkg_resources.py", line 1438, in run_script execfile(script_filename, namespace, namespace) File "/usr/lib/python3/dist-packages/pkg_resources.py", line 59, in execfile exec(compile(open(fn).read(), fn, 'exec'), globs, locs) File "/usr/local/lib/python3.4/dist-packages/opensipscli-0.1-py3.4.egg/EGG-INFO/scripts/opensips-cli", line 9, in run_console() File "/usr/local/lib/python3.4/dist-packages/opensipscli-0.1-py3.4.egg/EGG-INFO/scripts/opensips-cli", line 6, in run_console main.main() File "/usr/local/lib/python3.4/dist-packages/opensipscli-0.1-py3.4.egg/opensipscli/main.py", line 78, in main sys.exit(shell.cmdloop()) File "/usr/local/lib/python3.4/dist-packages/opensipscli-0.1-py3.4.egg/opensipscli/cli.py", line 225, in cmdloop ret = self.run_command(self.command[0], command, params) File "/usr/local/lib/python3.4/dist-packages/opensipscli-0.1-py3.4.egg/opensipscli/cli.py", line 331, in run_command return mod[0].invoke(cmd, params) File "/usr/local/lib/python3.4/dist-packages/opensipscli-0.1-py3.4.egg/opensipscli/modules/mi.py", line 126, in invoke res = comm.execute(cmd, params) File "/usr/local/lib/python3.4/dist-packages/opensipscli-0.1-py3.4.egg/opensipscli/comm.py", line 40, in execute ret = comm_handler.execute(cmd, params) File "/usr/local/lib/python3.4/dist-packages/opensipscli-0.1-py3.4.egg/opensipscli/communication/fifo.py", line 64, in execute replycmd = reply_fifo.readline() File "/usr/lib/python3.4/codecs.py", line 313, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 87-88: invalid continuation byte

Other commands are OK:

$ opensips-cli -x mi ul_show_contact location user1@test1.com SQLAlchemy version: 1.3.3 ERROR: command 'ul_show_contact' returned: 404: AOR not found

$ opensips-cli -x mi get_statistics tm: SQLAlchemy version: 1.3.3 { "tm:received_replies": 0, "tm:relayed_replies": 0, "tm:local_replies": 0, "tm:UAS_transactions": 0, "tm:UAC_transactions": 0, "tm:2xx_transactions": 0, "tm:3xx_transactions": 0, "tm:4xx_transactions": 0, "tm:5xx_transactions": 0, "tm:6xx_transactions": 0, "tm:inuse_transactions": 0 }

[MayamaTakeshi]

ul_dump works on other interfaces like mi_datagram. I tested with this script: https://pastebin.com/G7UEM9NK

I believe the problem is with the name of one of the usrloc domains:

$ ./ul_dump.py 
{"jsonrpc":"2.0","result":{"Domains":[{"name":"X","hash_size":512,"AORs":[]},{"name":"8^","hash_size":512,"AORs":[]},{"name":"location","hash_size":512,"AORs":[]}]},"id":1000}

$ ./ul_dump.py | jq .
{
  "jsonrpc": "2.0",
  "result": {
    "Domains": [
      {
        "name": "X",
        "hash_size": 512,
        "AORs": []
      },
      {
        "name": "8��^\u007f",
        "hash_size": 512,
        "AORs": []
      },
      {
        "name": "location",
        "hash_size": 512,
        "AORs": []
      }
    ]
  },
  "id": 1000
}

I am only using domain 'location'. I didn't create the other usrloc domains and I have no idea where they came from.

[MayamaTakeshi]

I am using Debian 8 and I cannot use latest commit because of problems to install pyOpenSSL on it. I am upgrading to Debian 9. I will report if the problem persists later.

[razvancrainea]

It looks like opensips-cli cannot decode the output returned by OpenSIPS properly due to the fact that OpenSIPS returns an invalid domain name. Could you tell us what are the domains you are using in your OpenSIPS script?

[MayamaTakeshi]

I upgraded to Debian9 and built latest commit of opensips-cli and confirmed the problem still exists:

$ cat /etc/issue
Debian GNU/Linux 9 \n \l

$ git log |head -n 1
commit 880e9a4bfbf4047fc6a53f39cb182b6b535e4555

$ opensips-cli -x mi ul_dump
Traceback (most recent call last):
  File "/usr/local/bin/opensips-cli", line 4, in <module>
    __import__('pkg_resources').run_script('opensipscli==0.1.0', 'opensips-cli')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 739, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1494, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.5/dist-packages/opensipscli-0.1.0-py3.5.egg/EGG-INFO/scripts/opensips-cli", line 9, in <module>
    run_console()
  File "/usr/local/lib/python3.5/dist-packages/opensipscli-0.1.0-py3.5.egg/EGG-INFO/scripts/opensips-cli", line 6, in run_console
    main.main()
  File "/usr/local/lib/python3.5/dist-packages/opensipscli-0.1.0-py3.5.egg/opensipscli/main.py", line 78, in main
    sys.exit(shell.cmdloop())
  File "/usr/local/lib/python3.5/dist-packages/opensipscli-0.1.0-py3.5.egg/opensipscli/cli.py", line 225, in cmdloop
    ret = self.run_command(self.command[0], command, params)
  File "/usr/local/lib/python3.5/dist-packages/opensipscli-0.1.0-py3.5.egg/opensipscli/cli.py", line 331, in run_command
    return mod[0].__invoke__(cmd, params)
  File "/usr/local/lib/python3.5/dist-packages/opensipscli-0.1.0-py3.5.egg/opensipscli/modules/mi.py", line 127, in __invoke__
    res = comm.execute(cmd, params)
  File "/usr/local/lib/python3.5/dist-packages/opensipscli-0.1.0-py3.5.egg/opensipscli/comm.py", line 40, in execute
    ret = comm_handler.execute(cmd, params)
  File "/usr/local/lib/python3.5/dist-packages/opensipscli-0.1.0-py3.5.egg/opensipscli/communication/fifo.py", line 64, in execute
    replycmd = reply_fifo.readline()
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 48: invalid start byte

$ ./ul_dump.py 
{"jsonrpc":"2.0","result":{"Domains":[{"name":"X)\u0011","hash_size":512,"AORs":[]},{"name":"8l)\u0011","hash_size":512,"AORs":[]},{"name":"location","hash_size":512,"AORs":[]}]},"id":1000}

$ ./ul_dump.py | jq .
{
  "jsonrpc": "2.0",
  "result": {
    "Domains": [
      {
        "name": "X�)\u0011�\u007f",
        "hash_size": 512,
        "AORs": []
      },
      {
        "name": "8l)\u0011�\u007f",
        "hash_size": 512,
        "AORs": []
      },
      {
        "name": "location",
        "hash_size": 512,
        "AORs": []
      }
    ]
  },
  "id": 1000
}

The only domain I am using in my opensips.cfg file is the "location". Maybe there is a bug in opensips and it is adding these 2 extra domains with name containing garbage.

[razvancrainea]

Yes, this is a bug in OpenSIPS, not OpenSIPS CLI. Please open a new ticket in the opensips issue tracker. In the meantime, I will keep this ticket open, because we need to add some extra checks to prevent OpenSIPS CLI from not dumping the output if something bogus is happening in OpenSIPS

[MayamaTakeshi]

I tested again this time by using sample etc/opensips.cfg file and it doesn't happen with it:

$ opensips-cli -x mi ul_dump
{
    "Domains": [
        {
            "name": "location",
            "hash_size": 512,
            "AORs": []
        }
    ]
}

So this might not be a bug with opensips after all and opensips-cli might not need adjustment (it might be something in my opensips.cfg). I will try to find which module or feature I use in my opensips.cfg that causes this and if it turns out to be a bug in opensips I'll open a ticket. Thanks.

[razvancrainea]

Well, it's definitely a problem in OpenSIPS, as there shouldn't be any extra domains rather than the ones used in the script. And in the script you are not allowed to add any non-printable domains names :). Could you please paste all the snippets that use save() and lookup() functions?

[MayamaTakeshi]

I have found the problem. It is not an opensips bug: we are using a custom module that does a special lookup into usrloc. For this, it registers a function that needs to do domain_fixup. This custom module was implemented for opensips-1.11 and domain_fixup was copied from registrar/reg_mod.c. So we updated the code for 3.0, however, in opensips-3.0 this function domain_fixup has changed and we didn't notice this change. So we didn't updated it so it was using old code for 1.11 and although it works, it caused the addition of those strange extra domains somehow. Sorry for taking your time with this. I'm closing this ticket.