ValueError: malformed node or string

[simonw]

Got this running against sqlite-utils:

symbex -s -d ../sqlite-utils

  File "/Users/simon/Dropbox/Development/symbex/symbex/lib.py", line 112, in <genexpr>
    defaults.extend(literal_eval(default) for default in function_node.args.defaults)
                    ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ast.py", line 110, in literal_eval
    return _convert(node_or_string)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ast.py", line 109, in _convert
    return _convert_signed_num(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ast.py", line 83, in _convert_signed_num
    return _convert_num(node)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ast.py", line 74, in _convert_num
    _raise_malformed_node(node)
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ast.py", line 71, in _raise_malformed_node
    raise ValueError(msg + f': {node!r}')
ValueError: malformed node or string on line 52: <ast.Name object at 0x101be12d0>

[simonw]

Cause is this line: https://github.com/simonw/symbex/blob/9d4170f8065809c1b94e54184918bd429ce8e7d1/symbex/lib.py#L112

[simonw]

Ran it in the debugger like this:

% python -i $(which symbex) -s -d ../sqlite-utils

import pdb; pdb.pm()

After some poking around:

<ast.FunctionDef object at 0x10519e470>
(Pdb) function_node.args.defaults
[<ast.Constant object at 0x10519fac0>, <ast.Name object at 0x10519faf0>]
(Pdb) function_node.args.defaults[0].__dict__
{'value': ',', 'kind': None, 'lineno': 52, 'col_offset': 31, 'end_lineno': 52, 'end_col_offset': 34}
(Pdb) function_node.args.defaults[1].__dict__
{'id': 'str', 'ctx': <ast.Load object at 0x1045f98a0>, 'lineno': 52, 'col_offset': 41, 'end_lineno': 52, 'end_col_offset': 44}
(Pdb) function_node.name
'jsonsplit'

So it's for this function here: https://github.com/simonw/sqlite-utils/blob/2747257a3334d55e890b40ec58fada57ae8cfbfd/sqlite_utils/recipes.py#L52

def jsonsplit(value, delimiter=",", type=str):
    """
    Convert a string like a,b,c into a JSON array ["a", "b", "c"]
    """
    return json.dumps([type(s.strip()) for s in value.split(delimiter)])

[simonw]

More debugging:

>>> import pdb; pdb.pm()
> /opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ast.py(71)_raise_malformed_node()
-> raise ValueError(msg + f': {node!r}')
(Pdb) u
> /opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ast.py(74)_convert_num()
-> _raise_malformed_node(node)
(Pdb) node
<ast.Name object at 0x1038a7a00>
(Pdb) node.__dict__
{'id': 'str', 'ctx': <ast.Load object at 0x102d018a0>, 'lineno': 52, 'col_offset': 41, 'end_lineno': 52, 'end_col_offset': 44}
(Pdb) ast
*** NameError: name 'ast' is not defined
(Pdb) import ast
(Pdb) ast.literal_eval(node)
*** ValueError: malformed node or string on line 52: <ast.Name object at 0x1038a7a00>

[simonw]

I'm going to catch that ValueError and try and show node.id instead, and fall back on ....

[simonw]

This seems to fix it, needs some tests though:

diff --git a/symbex/lib.py b/symbex/lib.py
index 38f3a1f..97aa5c7 100644
--- a/symbex/lib.py
+++ b/symbex/lib.py
@@ -109,7 +109,12 @@ def function_definition(function_node: AST):
     # if defaults has 2 and args has 3 then those
     # defaults correspond to the last two args
     defaults = [None] * (len(all_args) - len(function_node.args.defaults))
-    defaults.extend(literal_eval(default) for default in function_node.args.defaults)
+    for default in function_node.args.defaults:
+        try:
+            value = literal_eval(default)
+        except ValueError:
+            value = getattr(default, "id", "...")
+        defaults.append(value)

     arguments = []

[simonw]

Still needs tests.

[simonw]

Test with def jsonsplit(value, delimiter=",", type=str):.

[simonw]

Fixed:

symbex -d ../sqlite-utils/sqlite_utils -s 'json*'

# File: /Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py Line: 3625
def jsonify_if_needed(value)

# File: /Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py Line: 3117
def json_binary(value)

# File: /Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/recipes.py Line: 52
def jsonsplit(value, delimiter=",", type=str)

symbex -d ../sqlite-utils/sqlite_utils 'json*'   

# File: /Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py Line: 3625
def jsonify_if_needed(value):
    if isinstance(value, decimal.Decimal):
        return float(value)
    if isinstance(value, (dict, list, tuple)):
        return json.dumps(value, default=repr, ensure_ascii=False)
    elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):
        return value.isoformat()
    elif isinstance(value, uuid.UUID):
        return str(value)
    else:
        return value

# File: /Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py Line: 3117
def json_binary(value):
    if isinstance(value, bytes):
        return {"$base64": True, "encoded": base64.b64encode(value).decode("latin-1")}
    else:
        raise TypeError

# File: /Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/recipes.py Line: 52
def jsonsplit(value, delimiter=",", type=str):
    """
    Convert a string like a,b,c into a JSON array ["a", "b", "c"]
    """
    return json.dumps([type(s.strip()) for s in value.split(delimiter)])