numpy functions don't treat Any[...] arrays like Python lists

[stevengj]

A problem reported in https://github.com/JuliaPy/PythonPlot.jl/issues/36 — Numpy functions work for empty Python lists, and for empty numeric-typed Julia arrays, but don't work for empty Any[] arrays:

julia> using PythonCall

julia> np = pyimport("numpy")
Python: <module 'numpy' from '/Users/stevenj/.julia/environments/v1.10/.CondaPkg/env/lib/python3.12/site-packages/numpy/__init__.py'>

julia> np.isfinite(Int[])
Python: array([], dtype=bool)

julia> np.isfinite(pylist())
Python: array([], dtype=bool)

julia> np.isfinite([])
ERROR: Python: TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
Python stacktrace: none
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/wXfah/src/err.jl:94
 [2] errcheck
   @ ~/.julia/packages/PythonCall/wXfah/src/err.jl:10 [inlined]
 [3] pycallargs
   @ ~/.julia/packages/PythonCall/wXfah/src/abstract/object.jl:210 [inlined]
 [4] pycall(f::Py, args::Vector{Any}; kwargs::@Kwargs{})
   @ PythonCall ~/.julia/packages/PythonCall/wXfah/src/abstract/object.jl:228
 [5] pycall
   @ ~/.julia/packages/PythonCall/wXfah/src/abstract/object.jl:218 [inlined]
 [6] (::Py)(args::Vector{Any})
   @ PythonCall ~/.julia/packages/PythonCall/wXfah/src/Py.jl:341
 [7] top-level scope
   @ REPL[5]:1

Ideally, Py([]) should behave much like a python empty list. I'm not sure if this is something that can be fixed on the PythonCall side or if it's some broken special-casing in numpy?

The error message stems from linear_search_type_resolver in numpy

The basic issue seems to be that numpy treats an empty list as an array of float64, but Py([]) as an array of object:

julia> np.asarray(pylist())
Python: array([], dtype=float64)

julia> np.asarray([])
Python: array([], dtype=object)

The latter seems more correct, but I'm guessing that Numpy implemented the former rule for convenience. I can't find where the former rule is implemented in the numpy source, however, to tell if we can hook into it.

Not sure if this is something that can/should be changed, but I thought I should make a record of the problem for future reference.

[stevengj]

It's not just empty arrays:

julia> np.isfinite(pylist([1,2,3]))
Python: array([ True,  True,  True])

julia> np.isfinite(Any[1,2,3])
ERROR: Python: TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

julia> np.isfinite(np.asarray(pylist([1,2,3]), dtype=pybuiltins.object))
ERROR: Python: TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Basically, NumPy seems to treat Python lists fundamentally differently from numpy arrays of objects, and Py([]) acts like the latter. Seems like an upstream issue.

[stevengj]

I posted a question on https://mail.python.org/archives/list/numpy-discussion@python.org/ … we'll see what they say.