search

numba / numba

NumPy aware dynamic Python compiler using LLVM
https://numba.pydata.org/
BSD 2-Clause "Simplified" License
9.76k stars 1.12k forks source link

Default argument with typed list? #4382

Open sklam opened 5 years ago

sklam commented 5 years ago

(From gitter discussion: https://gitter.im/numba/numba?at=5d3f641b6009f73e06eb0f99)

using empty list as the default value to an argument is an interesting situation. First, numba doesn’t support the use of reflected-list or the new typed list as default argument value currently. Second, in python, default argument of list (or any mutable objects) can be easily misused; i.e.

In [1]: def foo(x=[]):
   ...:     x.append(1)
   ...:     return x
   ...:

In [2]: foo()
Out[2]: [1]

In [3]: foo()
Out[3]: [1, 1]
sklam commented 5 years ago

From a quick googling, the recommended way is:


In [1]: def foo(x=None):
   ...:     if x is None:
   ...:         x = []
   ...:     x.append(1)
   ...:     return x
   ...:

In [2]: foo()
Out[2]: [1]

In [3]: foo()
Out[3]: [1]

In [4]: foo([1,2])
Out[4]: [1, 2, 1]

But, numba does not handle it properly:


In [8]: from numba import njit, typed

In [9]: def foo(x=None):
   ...:     if x is None:
   ...:         x = typed.List()
   ...:     x.append(1)
   ...:     return x
   ...:

In [10]: foo()
Out[10]: ListType[int64]([1])

In [11]: cfoo = njit(foo)

In [12]: cfoo()
---------------------------------------------------------------------------
TypingError                               Traceback (most recent call last)
<ipython-input-12-f628c06b9d05> in <module>
----> 1 cfoo()

~/dev/numba/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    374                 e.patch_message(msg)
    375
--> 376             error_rewrite(e, 'typing')
    377         except errors.UnsupportedError as e:
    378             # Something unsupported is present in the user code, add help info

~/dev/numba/numba/dispatcher.py in error_rewrite(e, issue_type)
    341                 raise e
    342             else:
--> 343                 reraise(type(e), e, None)
    344
    345         argtypes = []

~/dev/numba/numba/six.py in reraise(tp, value, tb)
    656             value = tp()
    657         if value.__traceback__ is not tb:
--> 658             raise value.with_traceback(tb)
    659         raise value
    660

TypingError: Failed in nopython mode pipeline (step: nopython mode backend)
Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<function numba_typeref_ctor at 0x11d41a048>) with argument(s) of type(s): (typeref[OptionalType(ListType[int64]) i.e. the type 'ListType[int64] or None'])
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
In definition 2:
    All templates rejected with literals.
In definition 3:
    All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: resolving callee type: Function(<function numba_typeref_ctor at 0x11d41a048>)
[2] During: typing of call at /Users/siu/dev/numba/numba/targets/builtins.py (517)

File "numba/targets/builtins.py", line 517:
    def call_ctor(cls, *args):
        return numba_typeref_ctor(cls, *args)
        ^

[1] During: lowering "$8.3 = call $8.2(func=$8.2, args=[], kws=(), vararg=None)" at <ipython-input-9-549331c3ee61> (3)
This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new
smithseant commented 5 years ago

When using pure python I too follow the style recommendations for avoiding the mutable argument issue by: setting the default value to None, checking for that case, and re-setting the value to an empty list. But as I'm sure you are aware, numba doesn't ever allow changing the data type within the function. It seems the only current option is to create two separate functions. 

In [1]: from numba import __version__, jit, typed                                                   

In [2]: print(__version__)                                                                          
0.45.0

In [3]: @jit(nopython=True) 
   ...: def foo(x, y): 
   ...:     y.append(x) 
   ...:     return y 
   ...:                                                                                             

In [4]: a = typed.List()                                                                            

In [5]: a.append(42)                                                                                

In [6]: a.pop(0)                                                                                    
Out[6]: 42

In [7]: foo(7, a)                                                                                   
Out[7]: ListType[int64]([7])

In [8]: @jit(nopython=True) 
   ...: def bar(x): 
   ...:     y = typed.List() 
   ...:     y.append(42) 
   ...:     y.pop(0) 
   ...:     return foo(x, y) 
   ...:                                                                                             

In [9]: bar(7)                                                                                      
Out[9]: ListType[int64]([7])

...which may have been obvious to those to have more experience with typed lists.

stuartarchibald commented 5 years ago

I think branch pruning is also failing to spot that the branch can be removed and the block inlined in the default case.

sklam commented 5 years ago

So, there are two issues identified so far.