Unhandled Python exception raised calling 'execute' method

[vries]

With the master branch, and patch "gcc-with-cpychecker: Add CC_FOR_CPYCHECKER", I run make demo, and run into

CC_FOR_CPYCHECKER=/home/vries/python/tw-gcc7.sh LD_LIBRARY_PATH=gcc-c-api CC=/home/vries/python/tw-gcc7.sh ./gcc-with-cpychecker -c -I/usr/include/python2.7 -I/usr/include/python2.7 demo.c
demo.c: In function ‘socket_htons’:
demo.c:30:10: warning: Mismatching type in call to PyArg_ParseTuple with format code "i:htons"
     if (!PyArg_ParseTuple(args, "i:htons", &x1)) {
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  argument 3 ("&x1") had type
    "long unsigned int *" (pointing to 64 bits)
  but was expecting
    "int *" (pointing to 32 bits)
  for format code "i"
demo.c:30:10: error: Unhandled Python exception raised calling 'execute' method
Traceback (most recent call last):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 79, in execute
    self._check_refcounts(fun)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 85, in _check_refcounts
    dump_json=self.dump_json)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4393, in check_refcounts
    maxtrans)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4248, in impl_check_refcounts
    limits=limits)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 3033, in iter_traces
    depth + 1):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2997, in iter_traces
    transitions = curstate.get_transitions()
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2049, in get_transitions
    return self._get_transitions_for_stmt(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2065, in _get_transitions_for_stmt
    return self._get_transitions_for_GimpleCall(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2232, in _get_transitions_for_GimpleCall
    return meth(stmt, *args)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1161, in impl_PyArg_ParseTuple
    v_fmt, v_varargs, with_size_t=False)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1064, in _handle_PyArg_function
    s_success = self.state.mkstate_concrete_return_of(stmt, 1)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2116, in mkstate_concrete_return_of
    stmt.loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1567, in assign
    dest_region = self.eval_lvalue(lhs, loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1447, in eval_lvalue
    region = self.var_region(expr.var)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1576, in var_region
    check_isinstance(var, (gcc.VarDecl, gcc.ParmDecl, gcc.ResultDecl, gcc.FunctionDecl))
  File "/home/vries/python/gcc-python-plugin/gccutils/__init__.py", line 637, in check_isinstance
    raise TypeError('%s / %r is not an instance of %s' % (obj, obj, types))
TypeError: None / None is not an instance of (<type 'gcc.VarDecl'>, <type 'gcc.ParmDecl'>, <type 'gcc.ResultDecl'>, <type 'gcc.FunctionDecl'>)
demo.c: In function ‘not_enough_varargs’:
demo.c:40:9: warning: Not enough arguments in call to PyArg_ParseTuple with format string "i"
    if (!PyArg_ParseTuple(args, "i")) {
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
  expected 1 extra arguments:
    "int *" (pointing to 32 bits)
  but got none
demo.c:40:9: error: Unhandled Python exception raised calling 'execute' method
Traceback (most recent call last):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 79, in execute
    self._check_refcounts(fun)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 85, in _check_refcounts
    dump_json=self.dump_json)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4393, in check_refcounts
    maxtrans)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4248, in impl_check_refcounts
    limits=limits)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 3033, in iter_traces
    depth + 1):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2997, in iter_traces
    transitions = curstate.get_transitions()
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2049, in get_transitions
    return self._get_transitions_for_stmt(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2065, in _get_transitions_for_stmt
    return self._get_transitions_for_GimpleCall(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2232, in _get_transitions_for_GimpleCall
    return meth(stmt, *args)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1161, in impl_PyArg_ParseTuple
    v_fmt, v_varargs, with_size_t=False)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1064, in _handle_PyArg_function
    s_success = self.state.mkstate_concrete_return_of(stmt, 1)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2116, in mkstate_concrete_return_of
    stmt.loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1567, in assign
    dest_region = self.eval_lvalue(lhs, loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1447, in eval_lvalue
    region = self.var_region(expr.var)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1576, in var_region
    check_isinstance(var, (gcc.VarDecl, gcc.ParmDecl, gcc.ResultDecl, gcc.FunctionDecl))
  File "/home/vries/python/gcc-python-plugin/gccutils/__init__.py", line 637, in check_isinstance
    raise TypeError('%s / %r is not an instance of %s' % (obj, obj, types))
TypeError: None / None is not an instance of (<type 'gcc.VarDecl'>, <type 'gcc.ParmDecl'>, <type 'gcc.ResultDecl'>, <type 'gcc.FunctionDecl'>)
demo.c: In function ‘too_many_varargs’:
demo.c:50:10: warning: Too many arguments in call to PyArg_ParseTuple with format string "i"
     if (!PyArg_ParseTuple(args, "i", &i, &j)) {
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  expected 1 extra arguments:
    "int *" (pointing to 32 bits)
  but got 2:
    "int *" (pointing to 32 bits)
    "int *" (pointing to 32 bits)
demo.c:50:10: error: Unhandled Python exception raised calling 'execute' method
Traceback (most recent call last):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 79, in execute
    self._check_refcounts(fun)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 85, in _check_refcounts
    dump_json=self.dump_json)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4393, in check_refcounts
    maxtrans)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4248, in impl_check_refcounts
    limits=limits)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 3033, in iter_traces
    depth + 1):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2997, in iter_traces
    transitions = curstate.get_transitions()
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2049, in get_transitions
    return self._get_transitions_for_stmt(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2065, in _get_transitions_for_stmt
    return self._get_transitions_for_GimpleCall(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2232, in _get_transitions_for_GimpleCall
    return meth(stmt, *args)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1161, in impl_PyArg_ParseTuple
    v_fmt, v_varargs, with_size_t=False)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1064, in _handle_PyArg_function
    s_success = self.state.mkstate_concrete_return_of(stmt, 1)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2116, in mkstate_concrete_return_of
    stmt.loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1567, in assign
    dest_region = self.eval_lvalue(lhs, loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1447, in eval_lvalue
    region = self.var_region(expr.var)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1576, in var_region
    check_isinstance(var, (gcc.VarDecl, gcc.ParmDecl, gcc.ResultDecl, gcc.FunctionDecl))
  File "/home/vries/python/gcc-python-plugin/gccutils/__init__.py", line 637, in check_isinstance
    raise TypeError('%s / %r is not an instance of %s' % (obj, obj, types))
TypeError: None / None is not an instance of (<type 'gcc.VarDecl'>, <type 'gcc.ParmDecl'>, <type 'gcc.ResultDecl'>, <type 'gcc.FunctionDecl'>)
demo.c: In function ‘kwargs_example’:
demo.c:62:10: warning: Mismatching type in call to PyArg_ParseTupleAndKeywords with format code "(ff):kwargs_example"
     if (!PyArg_ParseTupleAndKeywords(args, kwargs, "(ff):kwargs_example", keywords, &x, &y)) {
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  argument 5 ("&x") had type
    "double *" (pointing to 64 bits)
  but was expecting
    "float *" (pointing to 32 bits)
  for format code "f"
demo.c:62:10: warning: Mismatching type in call to PyArg_ParseTupleAndKeywords with format code "(ff):kwargs_example"
  argument 6 ("&y") had type
    "double *" (pointing to 64 bits)
  but was expecting
    "float *" (pointing to 32 bits)
  for format code "f"
demo.c:62:10: error: Unhandled Python exception raised calling 'execute' method
Traceback (most recent call last):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 79, in execute
    self._check_refcounts(fun)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 85, in _check_refcounts
    dump_json=self.dump_json)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4393, in check_refcounts
    maxtrans)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4248, in impl_check_refcounts
    limits=limits)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 3033, in iter_traces
    depth + 1):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 3033, in iter_traces
    depth + 1):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 3033, in iter_traces
    depth + 1):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2997, in iter_traces
    transitions = curstate.get_transitions()
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2049, in get_transitions
    return self._get_transitions_for_stmt(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2065, in _get_transitions_for_stmt
    return self._get_transitions_for_GimpleCall(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2232, in _get_transitions_for_GimpleCall
    return meth(stmt, *args)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1186, in impl_PyArg_ParseTupleAndKeywords
    v_fmt, v_varargs, with_size_t=False)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1064, in _handle_PyArg_function
    s_success = self.state.mkstate_concrete_return_of(stmt, 1)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2116, in mkstate_concrete_return_of
    stmt.loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1567, in assign
    dest_region = self.eval_lvalue(lhs, loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1447, in eval_lvalue
    region = self.var_region(expr.var)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1576, in var_region
    check_isinstance(var, (gcc.VarDecl, gcc.ParmDecl, gcc.ResultDecl, gcc.FunctionDecl))
  File "/home/vries/python/gcc-python-plugin/gccutils/__init__.py", line 637, in check_isinstance
    raise TypeError('%s / %r is not an instance of %s' % (obj, obj, types))
TypeError: None / None is not an instance of (<type 'gcc.VarDecl'>, <type 'gcc.ParmDecl'>, <type 'gcc.ResultDecl'>, <type 'gcc.FunctionDecl'>)
demo.c: In function ‘buggy_converter’:
demo.c:76:10: warning: Mismatching type in call to PyArg_ParseTuple with format code "O&"
     if (!PyArg_ParseTuple(args, "O&", convert_to_ssize, &i)) {
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  argument 4 ("&i") had type
    "int *" (pointing to 32 bits)
  but was expecting
    "Py_ssize_t *" (pointing to 64 bits) (from second argument of "int (*fn) (struct PyObject *, Py_ssize_t *)")
  for format code "O&"
demo.c:76:10: error: Unhandled Python exception raised calling 'execute' method
Traceback (most recent call last):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 79, in execute
    self._check_refcounts(fun)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 85, in _check_refcounts
    dump_json=self.dump_json)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4393, in check_refcounts
    maxtrans)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4248, in impl_check_refcounts
    limits=limits)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 3033, in iter_traces
    depth + 1):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2997, in iter_traces
    transitions = curstate.get_transitions()
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2049, in get_transitions
    return self._get_transitions_for_stmt(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2065, in _get_transitions_for_stmt
    return self._get_transitions_for_GimpleCall(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2232, in _get_transitions_for_GimpleCall
    return meth(stmt, *args)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1161, in impl_PyArg_ParseTuple
    v_fmt, v_varargs, with_size_t=False)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1064, in _handle_PyArg_function
    s_success = self.state.mkstate_concrete_return_of(stmt, 1)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2116, in mkstate_concrete_return_of
    stmt.loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1567, in assign
    dest_region = self.eval_lvalue(lhs, loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1447, in eval_lvalue
    region = self.var_region(expr.var)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1576, in var_region
    check_isinstance(var, (gcc.VarDecl, gcc.ParmDecl, gcc.ResultDecl, gcc.FunctionDecl))
  File "/home/vries/python/gcc-python-plugin/gccutils/__init__.py", line 637, in check_isinstance
    raise TypeError('%s / %r is not an instance of %s' % (obj, obj, types))
TypeError: None / None is not an instance of (<type 'gcc.VarDecl'>, <type 'gcc.ParmDecl'>, <type 'gcc.ResultDecl'>, <type 'gcc.FunctionDecl'>)
demo.c: In function ‘make_a_list_of_random_ints_badly’:
demo.c:90:10: warning: Mismatching type in call to PyArg_ParseTuple with format code "i"
     if (!PyArg_ParseTuple(args, "i", &count)) {
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  argument 3 ("&count") had type
    "long int *" (pointing to 64 bits)
  but was expecting
    "int *" (pointing to 32 bits)
  for format code "i"
demo.c:90:10: error: Unhandled Python exception raised calling 'execute' method
Traceback (most recent call last):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 79, in execute
    self._check_refcounts(fun)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/__init__.py", line 85, in _check_refcounts
    dump_json=self.dump_json)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4393, in check_refcounts
    maxtrans)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 4248, in impl_check_refcounts
    limits=limits)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 3033, in iter_traces
    depth + 1):
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2997, in iter_traces
    transitions = curstate.get_transitions()
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2049, in get_transitions
    return self._get_transitions_for_stmt(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2065, in _get_transitions_for_stmt
    return self._get_transitions_for_GimpleCall(stmt)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2232, in _get_transitions_for_GimpleCall
    return meth(stmt, *args)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1161, in impl_PyArg_ParseTuple
    v_fmt, v_varargs, with_size_t=False)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/refcounts.py", line 1064, in _handle_PyArg_function
    s_success = self.state.mkstate_concrete_return_of(stmt, 1)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 2116, in mkstate_concrete_return_of
    stmt.loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1567, in assign
    dest_region = self.eval_lvalue(lhs, loc)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1447, in eval_lvalue
    region = self.var_region(expr.var)
  File "/home/vries/python/gcc-python-plugin/libcpychecker/absinterp.py", line 1576, in var_region
    check_isinstance(var, (gcc.VarDecl, gcc.ParmDecl, gcc.ResultDecl, gcc.FunctionDecl))
  File "/home/vries/python/gcc-python-plugin/gccutils/__init__.py", line 637, in check_isinstance
    raise TypeError('%s / %r is not an instance of %s' % (obj, obj, types))
TypeError: None / None is not an instance of (<type 'gcc.VarDecl'>, <type 'gcc.ParmDecl'>, <type 'gcc.ResultDecl'>, <type 'gcc.FunctionDecl'>)
make: *** [Makefile:291: demo] Error 1

[vries]

AFAIU, the CpyCheckerGimplePass is supposed to run before rewriting gimple into SSA:

    # Register our GCC passes:                                                                                    
    gimple_ps = CpyCheckerGimplePass(**kwargs)
    if 1:
        # non-SSA version:                                                                                        
        gimple_ps.register_before('*warn_function_return')

However, gcc 7 introduces an ssa-name (for call arguments that are calls themselves), so in fact CpyCheckerGimplePass can encounter ssa-names.

This cause a problem in refcounts verifying, when trying to determine the Region for an ssa-name, which has expr.var == None:

   def eval_lvalue(self, expr, loc):
        ...
        if isinstance(expr, gcc.SsaName):
            region = self.var_region(expr.var)
            check_isinstance(region, Region)
            return region

[vries]

Hmm, this could be the gcc-with-cpychecker version of:

# absinterp and thus the refcount-checker have bit-rotted:                                                        
if GCC_VERSION >= 7000:
    exclude_tests_below('tests/cpychecker/absinterp')
    exclude_tests_below('tests/cpychecker/refcounts')

[vries]

This ( https://github.com/davidmalcolm/gcc-python-plugin/pull/135 ) pull request runs the demo by default.

It ( https://travis-ci.org/davidmalcolm/gcc-python-plugin/builds/435599205?utm_source=github_status&utm_medium=notification ) fails for gcc-7 and gcc-8, but passes before gcc-6 and earlier.

[vries]

Fixed (well, worked around) in pull request ( https://github.com/davidmalcolm/gcc-python-plugin/pull/139 ) , by disabling verify_refcounting for gcc-7 and later.

[vries]

Fix (https://github.com/davidmalcolm/gcc-python-plugin/pull/140, "libcpychecker: Switch off verify_refcounting for gcc-7 and later") merged into master.

Closing issue.

[davidmalcolm]

Thanks for investigating this.