Numba type inference fails due to FindOverload not finding the right overload

[sudo-panda]

For functions that take in args that are aliases to basic types (such as size_t) FindOverload fails to find the correct overload. This is because numba extension tries to find an overload with long long or unsigned long long but the original function is of the type size_t.

Minimal reproducible example:

import numba
import cppyy

cppyy.cppdef("""
size_t square (size_t x) {
    return x * x;
}
""")

@numba.njit
def test_square():
     i = 0
     return cppyy.gbl.square(i)

print(test_square)

gives:

Traceback (most recent call last):
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/errors.py", line 823, in new_error_context
    yield
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typeinfer.py", line 578, in __call__
    self.resolve(typeinfer, typevars, fnty)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typeinfer.py", line 601, in resolve
    sig = typeinfer.resolve_call(fnty, pos_args, kw_args)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typeinfer.py", line 1555, in resolve_call
    return self.context.resolve_function_type(fnty, pos_args, kw_args)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typing/context.py", line 196, in resolve_function_type
    res = self._resolve_user_function_type(func, args, kws)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typing/context.py", line 248, in _resolve_user_function_type
    return func.get_call_type(self, args, kws)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/cppyy/numba_ext.py", line 128, in get_call_type
    ol = CppFunctionNumbaType(self._func.__overload__(*(numba2cpp(x) for x in args)), self._is_method)
LookupError: signature "long long" not found

A more interesting example would be:

import numba
import cppyy

cppyy.cppdef("""
#include<vector>

std::vector<float> v = {1, 2, 3, 4, 5};
""")

@numba.njit
def test_vector_at(v):
     i = 0
     return v.at(i)

print(test_vector_at(cppyy.gbl.v))

[wlav]

Yes, finding correct (non-string based, or at least on resolved types) matching isn't available yet.

[sudo-panda]

So how do you suggest we make the numba extension support it?

Also accessing vectors by index:

import numba
import cppyy

cppyy.cppdef("""
#include<vector>

std::vector<float> v = {1, 2, 3, 4, 5};
""")

@numba.njit
def test_vector_at(v):
     i = 0
     return v[i]

print(test_vector_at(cppyy.gbl.v))

gives this error:

Traceback (most recent call last):
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typeinfer.py", line 155, in propagate
    constraint(typeinfer)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typeinfer.py", line 481, in __call__
    self.fallback(typeinfer)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typeinfer.py", line 704, in __call__
    self.resolve(typeinfer, typeinfer.typevars, fnty=fnty)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typeinfer.py", line 601, in resolve
    sig = typeinfer.resolve_call(fnty, pos_args, kw_args)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typeinfer.py", line 1555, in resolve_call
    return self.context.resolve_function_type(fnty, pos_args, kw_args)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typing/context.py", line 213, in resolve_function_type
    raise last_exception
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typing/context.py", line 196, in resolve_function_type
    res = self._resolve_user_function_type(func, args, kws)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/typing/context.py", line 248, in _resolve_user_function_type
    return func.get_call_type(self, args, kws)
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/types/functions.py", line 331, in get_call_type
    failures.raise_error()
  File "/home/sudo-panda/Documents/numba/.venv/lib/python3.10/site-packages/numba/core/types/functions.py", line 227, in raise_error
    raise errors.TypingError(self.format())
numba.core.errors.TypingError: No implementation of function Function(<built-in function getitem>) found for signature:

 >>> getitem(CppClass(std::vector<float>), Literal[int](0))

There are 22 candidate implementations:
  - Of which 22 did not match due to:
  Overload of function 'getitem': File: <numerous>: Line N/A.
    With argument(s): '(CppClass(std::vector<float>), int64)':
   No match.

I am not sure where to look for this.

[wlav]

Right now, the signature match is purely string based, see: https://github.com/wlav/CPyCppyy/blob/master/src/CPPOverload.cxx#L1085

This will have to be smarter, e.g. by resolving all typedefs first, or by finding the decls and comparing on them.

For vector, the problem is most likely that __getitem__ is Pythonized and therefore not a CPPOverload, making that it have no __overload__ to find the proper overload. Possible solutions: the Numba extension need to learn about pythonizations, pythonizations need to grow __overload__, or pythonizations need to have a custom variable to find the original CPPOverload. I don't think any solution can be 100% generalized as pythonizations are of course free form.

[sudo-panda]

Right now, the signature match is purely string based, see: https://github.com/wlav/CPyCppyy/blob/master/src/CPPOverload.cxx#L1085

Yeah I have seen that but can we have a hack that makes it possible for the numba extension to give __overload__ the correct argument, such as numba2cpp providing multiple cpp_types for the same numba type.

For vector, the problem is most likely that __getitem__ is Pythonized and therefore not a CPPOverload, making that it have no __overload__ to find the proper overload. Possible solutions: the Numba extension need to learn about pythonizations, pythonizations need to grow __overload__, or pythonizations need to have a custom variable to find the original CPPOverload. I don't think any solution can be 100% generalized as pythonizations are of course free form.

Alright I will look into them!

[wlav]

Not following the first: Numba will ever provide only one type? Also note that __overload__ is an old function, so it's original behavior needs to be preserved. Probably simple enough, though, to do a check on the argument type: if string, do original signature matching, if list, do Numba type matching, which should probably live in the backend if based on decls. (We can create a numba2decl on the Python side to avoid needing intermediate strings, but maybe numba2ir works as well.)

[wlav]

This was fixed somewhere along the way. :) Either your own matching code or Aaron's updates to it. Regardless, it now runs with cppyy 3.1.1, so closing.