Closed mgorny closed 9 months ago
Sorry T_T I forgot to call test(). I'll remember to do this afterwards.
My testing will output ImportError
like this:
Traceback (most recent call last):
File "/Users/liuteng/opt/anaconda3/envs/test_ne/lib/python3.9/site-packages/numexpr-2.8.8.dev1-py3.9-macosx-10.9-x86_64.egg/numexpr/__init__.py", line 64, in test
import numexpr.tests
File "/Users/liuteng/opt/anaconda3/envs/test_ne/lib/python3.9/site-packages/numexpr-2.8.8.dev1-py3.9-macosx-10.9-x86_64.egg/numexpr/tests/__init__.py", line 11, in <module>
from numexpr.tests.test_numexpr import test, print_versions
File "/Users/liuteng/opt/anaconda3/envs/test_ne/lib/python3.9/site-packages/numexpr-2.8.8.dev1-py3.9-macosx-10.9-x86_64.egg/numexpr/tests/test_numexpr.py", line 21, in <module>
from numpy import (
ImportError: cannot import name 'complex_' from 'numpy' (/Users/liuteng/opt/anaconda3/envs/test_ne/lib/python3.9/site-packages/numpy/__init__.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/liuteng/opt/anaconda3/envs/test_ne/lib/python3.9/site-packages/numexpr-2.8.8.dev1-py3.9-macosx-10.9-x86_64.egg/numexpr/__init__.py", line 68, in test
raise ImportError('`numexpr.tests` could not be imported, likely it was excluded from the distribution.')
ImportError: `numexpr.tests` could not be imported, likely it was excluded from the distribution
I found out that this is caused by importing complex_
from numpy
, while numpy
doesn't seems to have complex_
so I changed this to cdouble
and all tests passed.
I don't know why numexpr tries to import complex_
, I need to ask in numpy maillist about what complex_
was.
I'm completely new to pypy
, I will try to test it today.
I think you're testing PyPy3.9; I don't know how well that works with NumPy. We've only tried PyPy3.10.
I think you're testing PyPy3.9; I don't know how well that works with NumPy. We've only tried PyPy3.10.
I'm using CPython 3.9 and I'm figuring out how to install pypy T_T
If your distribution doesn't include packages for PyPy, I suppose you can download and run the precompiled version from https://www.pypy.org/download.html. Or possibly you could try a nightly from https://buildbot.pypy.org/nightly/py3.10/. They all can be run straight after unpacking, without having to "install" them in any way.
If your distribution doesn't include packages for PyPy, I suppose you can download and run the precompiled version from https://www.pypy.org/download.html. Or possibly you could try a nightly from https://buildbot.pypy.org/nightly/py3.10/. They all can be run straight after unpacking, without having to "install" them in any way.
Thanks, I'm using mac m1 and now it's working using the precompiled version.
Another stupid question is why can't I import numpy
and numexpr
while using pypy? Do I need to run pypy pip install numpy
to make numpy available for pypy?
Yes. pypy -m pip install numpy
.
It's continuously saying xxxx.so can't be opened because Apple cannot check it for malicious software
, maybe I should try on a Windows machine
I'm sorry but I really have no clue about Apple.
np, I have another windows machine😀
Michał Górny @.***> 于2023年12月2日周六 14:20写道:
I'm sorry but I really have no clue about Apple.
— Reply to this email directly, view it on GitHub https://github.com/pydata/numexpr/issues/463#issuecomment-1837147605, or unsubscribe https://github.com/notifications/unsubscribe-auth/A33BDH3XHRNFPGGZFP5LS7LYHMTKLAVCNFSM6AAAAABADLN5FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZXGE2DONRQGU . You are receiving this because you commented.Message ID: @.***>
I've finally replicated this error on Fedora with pypy 3.9, numexpr 2.8, numpy 1.26, I will try to fix this.
Ah, I seem to find the problem.
When we run in Python:
func = Numexpr(...)
func(...)
It will actually invoke interpreter.cpp:NumExpr_Run(self, args, kwds)
, and the third argument kwds
is a pointer to PyDict given to func
.
In CPython, if we don't provide any python dictionary to func()
, then the pointer kwds
will be 0 (or NULL, nullptr); while in pypy, I found out that this pointer is not NULL and there is nothing inside this "dictionary".
I don't quite understand why this will occur because I don't have a comprehensive understanding of how Python binds to C++, but I guess this is something wrong while calling C++ function, it's like the default argument is not 0 in pypy, but it is 0 in CPython.
But this could be fixed in numexpr
side anyways. I will think of a reasonable way.
It is probably a bug in PyPy. Could you help me replicate it so I can fix it? Is there a smaller reproducer than a complete Numexpr build?
It is probably a bug in PyPy. Could you help me replicate it so I can fix it? Is there a smaller reproducer than a complete Numexpr build?
It don't need to build numexpr to replicate it, using pypy and running numexpr.test()
will give this error.
Since this involves how python calls C/C++ extension, I will learn how to use setuptools
first and give you a small reproducer asap
By the way, is this behaviour considered a bug? I found that although kwds
is not a null pointer, but when I run Py_DictSize
it returns 0
so it's like when not given a dictionary, pypy will provide an empty dictionary instead of a nullptr, I don't know whether this is a bug or intended behaviour.
It is probably a bug in PyPy. Could you help me replicate it so I can fix it? Is there a smaller reproducer than a complete Numexpr build?
So sorry, I don't know how replicate this error with a simple C extension. I tried to write a very simple python extension foo
with one function bar
and pypy
works perfectly in this case, when I don't provide dictionary argument to function call, the dictionary pointer kwds
is NULL.
Anyways, I should fix it in the numexpr
side at first, I will inform you if I figure out how to write a small reproducer.
And may I ask another question, there is still an error says "AttributeError: module 'sys' has no attribute 'getrefcount'", and I tried this snippet:
import sys
a = [1]
print(sys.getrefcount(a))
And it gets the same error, but according to the Python standard this should be a valid attribute within module sys
. https://docs.python.org/3.9/library/sys.html#sys.getrefcount
Is this a bug or intended not to implement, or I missed something?
Is this a bug or intended not to implement
It is intentional. PyPy does not use a refcounting garbage collector. You are correct that it is hard to find information about that difference, I will add it to the garbage collection strategies section about differences between PyPy and CPython.
It's fixed by https://foss.heptapod.net/pypy/pypy/-/commit/402204e196ac7a86975d8d38f6abf2c384f1dd9d, should be closed now.
Thanks to you both!
I can still reproduce the problem with the newest PyPy3.10 nightly.
@mgorny you are still seeing ValueError: ex_uses_vml parameter is required
?
Yes:
$ curl https://buildbot.pypy.org/nightly/py3.10/pypy-c-jit-184208-0c87577e022f-linux64.tar.bz2 | tar -xj
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 31.3M 100 31.3M 0 0 3305k 0 0:00:09 0:00:09 --:--:-- 3358k
$ git clone https://github.com/pydata/numexpr
Cloning into 'numexpr'...
remote: Enumerating objects: 4682, done.
remote: Counting objects: 100% (731/731), done.
remote: Compressing objects: 100% (287/287), done.
remote: Total 4682 (delta 402), reused 660 (delta 363), pack-reused 3951
Receiving objects: 100% (4682/4682), 2.72 MiB | 13.12 MiB/s, done.
Resolving deltas: 100% (3085/3085), done.
$ /tmp/pypy-c-jit-184208-0c87577e022f-linux64/bin/pypy3 -m venv venv
$ . venv/bin/activate
$ pip install ./numexpr/
Processing ./numexpr
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting numpy>=1.13.3
Using cached numpy-1.26.3-pp310-pypy310_pp73-linux_x86_64.whl
Building wheels for collected packages: numexpr
Building wheel for numexpr (pyproject.toml) ... done
Created wheel for numexpr: filename=numexpr-2.8.9.dev1-pp310-pypy310_pp73-linux_x86_64.whl size=100692 sha256=9959c020ee42b83e0eab8e0
30f4a157fc3bb7cfedff70b18b10707e7a7125913
Stored in directory: /tmp/pip-ephem-wheel-cache-0ce9li2k/wheels/6a/90/d2/38a94db313dad36daf3e688b0f11f9019e3d64d3709d562cac
Successfully built numexpr
Installing collected packages: numpy, numexpr
Successfully installed numexpr-2.8.9.dev1 numpy-1.26.3
WARNING: There was an error checking the latest version of pip.
(venv) mgorny@pomiot /tmp $ pypy
Python 3.10.13 (0c87577e022f, Jan 04 2024, 10:42:56)
[PyPy 7.3.15-alpha0 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``snow, snow, snow, snow''
>>>> import numexpr
>>>> numexpr.test()
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Numexpr version: 2.8.9.dev1
NumPy version: 1.26.3
Python version: 3.10.13 (0c87577e022f, Jan 04 2024, 10:42:56)
[PyPy 7.3.15-alpha0 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
Platform: linux-x86_64-#1 SMP PREEMPT_DYNAMIC Mon Jan 1 16:05:17 -00 2024
CPU vendor:
CPU model:
CPU clock speed: MHz
VML available? False
Number of threads used by default: 8 (out of 12 detected cores)
Maximum number of threads: 64
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
....E.EEEE...............E.EEEE...........E.E...................E......................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.......................................................................................................................................
.
======================================================================
ERROR: test_rational_expr (numexpr.tests.test_numexpr.test_numexpr)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 76, in test_rational_expr
y = func(a, b)
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_refcount (numexpr.tests.test_numexpr.test_numexpr)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 314, in test_refcount
assert sys.getrefcount(a) == 2
AttributeError: module 'sys' has no attribute 'getrefcount'
======================================================================
ERROR: test_simple (numexpr.tests.test_numexpr.test_numexpr)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 56, in test_simple
x = func(array([1., 2, 3]), array([4., 5, 6]), array([7., 8, 9]))
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_simple_expr (numexpr.tests.test_numexpr.test_numexpr)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 68, in test_simple_expr
y = func(x)
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_simple_expr_small_array (numexpr.tests.test_numexpr.test_numexpr)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 62, in test_simple_expr_small_array
y = func(x)
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_rational_expr (numexpr.tests.test_numexpr.test_numexpr2)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 76, in test_rational_expr
y = func(a, b)
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_refcount (numexpr.tests.test_numexpr.test_numexpr2)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 314, in test_refcount
assert sys.getrefcount(a) == 2
AttributeError: module 'sys' has no attribute 'getrefcount'
======================================================================
ERROR: test_simple (numexpr.tests.test_numexpr.test_numexpr2)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 56, in test_simple
x = func(array([1., 2, 3]), array([4., 5, 6]), array([7., 8, 9]))
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_simple_expr (numexpr.tests.test_numexpr.test_numexpr2)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 68, in test_simple_expr
y = func(x)
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_simple_expr_small_array (numexpr.tests.test_numexpr.test_numexpr2)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 62, in test_simple_expr_small_array
y = func(x)
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_all_scalar (numexpr.tests.test_numexpr.test_evaluate)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 495, in test_all_scalar
assert_equal(expr(a, b), 2 * a + 3 * b)
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_broadcasting (numexpr.tests.test_numexpr.test_evaluate)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 488, in test_broadcasting
assert_array_equal(expr(a, c), 2.0 * a + 3.0 * c)
ValueError: ex_uses_vml parameter is required
======================================================================
ERROR: test_run (numexpr.tests.test_numexpr.test_evaluate)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/venv/lib/pypy3.10/site-packages/numexpr/tests/test_numexpr.py", line 501, in test_run
assert_array_equal(expr(a, b), expr.run(a, b))
ValueError: ex_uses_vml parameter is required
----------------------------------------------------------------------
Ran 5536 tests in 11.023s
FAILED (errors=13)
<unittest.runner.TextTestResult run=5536 errors=13 failures=0>
>>>>
So maybe PyPy still needs the fix from #467?
So maybe PyPy still needs the fix from #467?
Yes, that would help. That still leaves failures from test_refcount
but that's something that definitely needs to be fixed in numexpr (by skipping the test, I guess).
I see your reproducer is using numpy-1.26.3-pp310-pypy310_pp73-linux_x86_64.whl
. Does the problem still occur if you build numpy from source?
Yes, though I've used 7.3.14 and not nightly. Do you need me to retry with nightly?
No, there is no real difference between 7.3.14 and nightly.
Could this be closed with latests merged PRs?
Yes, I've just confirmed that all tests pass now.
When running the test suite on PyPy3.10 7.3.13, I'm getting the following failures:
This is on 05bb401afc1f8fd84e45f0297d3a0b87e6a79b30 but I've seen roughly the same result on 2.8.7.
CC @mattip