pydata / numexpr

Fast numerical array expression evaluator for Python, NumPy, Pandas, PyTables and more
https://numexpr.readthedocs.io/en/latest/user_guide.html
MIT License
2.23k stars 210 forks source link

FIX: do not use deprecated numpy flag #406

Closed tacaswell closed 2 years ago

tacaswell commented 2 years ago

Per https://numpy.org/doc/stable/reference/c-api/array.html#c.NPY_ARRAY_WRITEBACKIFCOPY and https://numpy.org/doc/stable/reference/c-api/array.html#c.NPY_ARRAY_UPDATEIFCOPY NPY_ARRAY_WRITEBACKIFCOPY is preferred.

tacaswell commented 2 years ago

Due to 55765c6e48dba994d1afe04e676e4a79a4229efc numexpr was not compiling on py311 + numpy main branch. I did not sort out which was the source of the problem, but I suspect numpy.

tacaswell commented 2 years ago

Appears to pass on my fork: https://github.com/tacaswell/numexpr/runs/6104048205

robbmcleod commented 2 years ago

Does this resolve #397? Can you try the code fragment in my message from Jan 24?

 x = np.empty(5, dtype=np.uint8)[1:].view(np.int32)
 print(f"pre-eval, x = {x} at {x.__array_interface__['data'][0]}")
 ne.evaluate('3', out=x)
 print(f"post-eval, x = {x} at {x.__array_interface__['data'][0]}")

and see if it actually sets x to 3?

tacaswell commented 2 years ago

Maybe?

In [4]:  x = np.empty(5, dtype=np.uint8)[1:].view(np.int32)
   ...:  print(f"pre-eval, x = {x} at {x.__array_interface__['data'][0]}")
   ...:  ne.evaluate('3', out=x)
   ...:  print(f"post-eval, x = {x} at {x.__array_interface__['data'][0]}")
pre-eval, x = [944689022] at 94820750688081
<ipython-input-4-422c0a4d6701>:3: RuntimeWarning: WRITEBACKIFCOPY detected in array_dealloc.  Required call to PyArray_ResolveWritebackIfCopy or PyArray_DiscardWritebackIfCopy is missing.
  ne.evaluate('3', out=x)
post-eval, x = [3] at 94820750688081

It seems to set x -> 3, but is producing a warning.

robbmcleod commented 2 years ago

Right, this is the same problem I had. We cannot know when to call PyArray_DiscardWritebackIfCopy because NumExpr doesn't own the data. Hence my thought that we should simply do the write immediately since this is such an edge case. You have to simultaneously be setting an array to a const, and provide the output array, to reach this branch.

I'm away this weekend, will try and look at it next week.

robbmcleod commented 2 years ago

Thanks for the effort but we went with a slightly different approach here.

tacaswell commented 2 years ago

No worries! So long as it got fixed I'm happy :)