HTTP Bridge cannot handle mixed type key values

[pogojotz]

Using different type of key values raises a TypeError in HTTP Bridge. E.g.:

Traceback (most recent call last):
  File "/home/jotz/dev/netcall-webkit/env/lib/python3.10/site-packages/txaio/tx.py", line 366, in as_future
    return maybeDeferred(fun, *args, **kwargs)
  File "/home/jotz/dev/netcall-webkit/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 167, in maybeDeferred
    result = f(*args, **kw)
  File "/home/jotz/dev/netcall-webkit/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1656, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/home/jotz/dev/netcall-webkit/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1571, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/home/jotz/dev/netcall-webkit/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1445, in _inlineCallbacks
    result = current_context.run(g.send, result)
  File "/home/jotz/dev/netcall-webkit/env/lib/python3.10/site-packages/crossbar/bridge/rest/subscriber.py", line 74, in on_event
    body = json.dumps(
  File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
builtins.TypeError: '<' not supported between instances of 'str' and 'int'

This is due to the use of sort_keys=True in the call to json.dumps() in crossbar/bridge/rest/subscriber.py.

This is both a bug report and feature request: sort_keys should be optional.

    :::::::::::::::::
          :::::          _____                      __
    :::::   :   :::::   / ___/____ ___   ___  ___  / /  ___ _ ____
    :::::::   :::::::  / /__ / __// _ \ (_-< (_-< / _ \/ _ `// __/
    :::::   :   :::::  \___//_/   \___//___//___//_.__/\_,_//_/
          :::::
    :::::::::::::::::   Crossbar v21.3.1

    Copyright (c) 2013-2022 Crossbar.io Technologies GmbH, licensed under AGPL 3.0.

 Crossbar.io        : 21.3.1
   txaio            : 21.2.1
   Autobahn         : 21.3.1
     UTF8 Validator : autobahn
     XOR Masker     : autobahn
     JSON Codec     : stdlib
     MsgPack Codec  : msgpack-1.0.2
     CBOR Codec     : cbor-1.0.0
     UBJSON Codec   : ubjson-0.16.1
     FlatBuffers    : flatbuffers-1.12
   Twisted          : 21.2.0-EPollReactor
   LMDB             : 1.2.1/lmdb-0.9.29
   Python           : 3.10.4/CPython
   PIP              : 21.3.1
 Frozen executable  : no
 Operating system   : Linux-5.15.32-1-lts-x86_64-with-glibc2.35
 Host machine       : x86_64
 Release key        : RWT906S6dtgOY3NZrHd3uFtXpAPFUc6+bdpADYJRJi4daUaCT5U/0OrO

[pogojotz]

1983

[oberstet]

Is the dict with the keys of mixed type the kwargs argument of the Event itself, or is the dict nested within the args or kwargs payload?

Also, just a remark, it is quite surprising to find such a bug (IMO) in the Python stdlib JSON serializer:

(cpy39_1) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar-examples/proxy$ python
Python 3.9.4 (default, Apr 29 2021, 02:30:43) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> d = {'a': 1, 2: 3}
>>> json.dumps(d, sort_keys=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/oberstet/cpy39/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "/home/oberstet/cpy39/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/oberstet/cpy39/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
TypeError: '<' not supported between instances of 'int' and 'str'
>>> 

another comment: as an alternative to the change you propose in the PR, we could also just do sort_keys=False. this wouldn't introduce a new, obscure option to the outside world (obscure, because sorting keys in JSON is undefined as keys can be mixed type, and there is no total order on that)

[pogojotz]

That would be even better! I just didn't know if there are any dependencies on this as a default behavior, e.g. CI, unittests, etc.

Speaking of which, I think the option sort_keys is exactly meant for test cases. From the Python doc:

If sort_keys is true (default: False), then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

Feel free to throw away my PR and implement as you suggested.

[pogojotz]

I just remembered, you asked a question :)
One of my kwargs-values is a dict and there keys are of different types.

[oberstet]

One of my kwargs-values is a dict and there keys are of different types.

thanks! (I was asking because the keys of kwargs MUST all be strings due to WAMP spec - but the keys in nested values are not .. which is your situation)

Feel free to throw away my PR and implement as you suggested.

I'll do. fwiw, I'll look at the stuff below, and piggy back it on my upcoming work PR .ö

(cpy39_1) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ find crossbar -name "*.py" -exec grep -Hi "sort_keys=True" {} \;
crossbar/shell/app.py:            json_str = json.dumps(result.result, separators=(', ', ': '), sort_keys=True, indent=4, ensure_ascii=False)
crossbar/_util.py:        return json.dumps(obj, indent=4, separators=(',', ': '), sort_keys=True, ensure_ascii=False)
crossbar/bridge/rest/subscriber.py:                              sort_keys=True,
crossbar/bridge/mqtt/test/test_wamp.py:                    payload=json.dumps({}, sort_keys=True).encode('utf8')).serialise())
crossbar/common/checkconfig.py:                    json.dump(config, outfile, ensure_ascii=False, separators=(',', ': '), indent=3, sort_keys=True)
crossbar/webservice/misc.py:            self._data = json.dumps(value, sort_keys=True, indent=3, ensure_ascii=False)
(cpy39_1) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$