Open EricCousineau-TRI opened 3 years ago
Already spotted this in #2768 as well, but haven't had time to look closer. We might not want to wait until 3.10 is released to report this (ahem we'd never let such a thing happen cough) ;-)
Before looking at the code, this looked like something is being passed through as a type annotation and being called, but now type annotations are all strings, so it's now calling a string instead of a callable. But have no clue where anything like that would be in our code. It seems to possibly be happening when a module is being reloaded? That could be an issue somewhere in Python's stdlib code.
I think 3.10a4 is the first with "from __future__ import annotations
" on by default, which is also suspicious.
We don't run these tests on Windows after 3.7 due to the DLL loading changes which we've never worked around, IIRC. So it's likely a problem everywhere.
We don't run these tests on Windows after 3.7 due to the DLL loading changes which we've never worked around, IIRC. So it's likely a problem everywhere.
Er, I understood your prior post, but I'm not sure how Windows 3.7 w/ DLL relates to 3.10-dev - can you expand on that?
Oh - meaning I should remove the (ubuntu, mac)
specifier. Derp.
We don't run these tests on Windows after 3.7 due to the DLL loading changes which we've never worked around, IIRC. So it's likely a problem everywhere.
In 3.8+ loading DLLs on Winblows changes in this way (os.add_dll_directory
):
https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-python-api https://docs.python.org/3/whatsnew/3.8.html#os https://docs.python.org/3/whatsnew/3.8.html#ctypes https://docs.python.org/3/library/os.html?highlight=add_dll_directory#os.add_dll_directory
Here's add_dll_directory
used in my project: https://github.com/ycm-core/ycmd/pull/1400/commits/6c28224b5ff5dca3cf1d72c02615c2da317ba4c6
@henryiii tried to fix this for pybind11, but with no success.
this looked like something is being passed through as a type annotation and being called, but now type annotations are all strings
That would mean this would be reproducible on python 3.9.1 with from __future__ import annotations
. I tried adding that line to all tests/**/test_*.py
files, but make cpptest
was successful
I just tried, but I can't reproduce locally with the last version of Python. I installed "Python 3.10.0a4+ (heads/master:d16f617, Jan 9 2021, 20:40:43)" through pyenv, and all tests pass. I'll give the exact same version, 3.10.0a4, a try as well.
I just tried, but I can't reproduce locally with the last version of Python. I installed "Python 3.10.0a4+ (heads/master:d16f617, Jan 9 2021, 20:40:43)" through pyenv, and all tests pass. I'll give the exact same version, 3.10.0a4, a try as well.
Scratch that. It's the embedding tests that fail. If you run those, CPython's master
still fails.
I suspect it's loading a file somewhere in Python, not our .py's. We don't use annotations, sadly, much less try to call them.
Is there some way to get more information about where the failure actually is, perhaps? If we just had a filename, that would be huge. We need from rich.traceback import install; install(show_locals=True)
, perhaps :)
Seems like something worse is going on, though, I'm afraid. If I comment out preceding tests in test_interpreter.cpp
, I'm getting a beautiful SIGABRT
:
Fatal Python error: compiler_make_closure: lookup '__class__' in DerivedWidget 5 -1
freevars of __init__: ('__class__',)
, globals=, locals=locals@entry=0x0, args=args@entry=0x7ffff5a5c3c8, argcount=4, kwnames=0x7ffff7ebad38,
kwargs=0x7ffff5a5c3e8, kwcount=2, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name='_call_with_frames_removed', qualname='_call_with_frames_removed') at ../Python/ceval.c:4620
#25 0x00007ffff7699de2 in _PyFunction_Vectorcall (func=, stack=0x7ffff5a5c3c8, nargsf=, kwnames=) at ../Objects/call.c:396
#26 0x00007ffff77a557a in _PyObject_VectorcallTstate (kwnames=('dont_inherit', 'optimize'), nargsf=9223372036854775812, args=0x7ffff5a5c3c8, callable=,
tstate=0x5555558154b0) at ../Include/cpython/abstract.h:114
#27 PyObject_Vectorcall (kwnames=('dont_inherit', 'optimize'), nargsf=9223372036854775812, args=0x7ffff5a5c3c8, callable=) at ../Include/cpython/abstract.h:123
#28 call_function (tstate=tstate@entry=0x5555558154b0, pp_stack=pp_stack@entry=0x7fffffffbe68, oparg=oparg@entry=6, kwnames=kwnames@entry=('dont_inherit', 'optimize')) at ../Python/ceval.c:5361
#29 0x00007ffff77b0b57 in _PyEval_EvalFrameDefault (tstate=0x5555558154b0,
--Type for more, q to quit, c to continue without paging--
f=Frame 0x7ffff5a5c240, for file , line 897, in source_to_code (self=, data=b'# -*- coding: utf-8 -*-\nfrom widget_module import Widget\n\n\nclass DerivedWidget(Widget):\n def __init__(self, message):\n super(DerivedWidget, self).__init__(message)\n\n def the_answer(self):\n return 42\n', path='./test_interpreter.py', _optimize=-1), throwflag=) at ../Python/ceval.c:3815
#30 0x00007ffff77b2e7e in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7ffff5a5c240, for file , line 897, in source_to_code (self=, data=b'# -*- coding: utf-8 -*-\nfrom widget_module import Widget\n\n\nclass DerivedWidget(Widget):\n def __init__(self, message):\n super(DerivedWidget, self).__init__(message)\n\n def the_answer(self):\n return 42\n', path='./test_interpreter.py', _optimize=-1), tstate=0x5555558154b0) at ../Include/internal/pycore_ceval.h:40
#31 _PyEval_EvalCode (tstate=0x5555558154b0, _co=, globals=, locals=locals@entry=0x0, args=args@entry=0x555555884ad0, argcount=3, kwnames=0x0,
kwargs=0x555555884ae8, kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs={'_optimize': -1}, closure=0x0, name='source_to_code', qualname='SourceLoader.source_to_code') at ../Python/ceval.c:4620
#32 0x00007ffff7699de2 in _PyFunction_Vectorcall (func=, stack=0x555555884ad0, nargsf=, kwnames=) at ../Objects/call.c:396
#33 0x00007ffff77a557a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775811, args=0x555555884ad0, callable=, tstate=0x5555558154b0)
at ../Include/cpython/abstract.h:114
#34 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775811, args=0x555555884ad0, callable=) at ../Include/cpython/abstract.h:123
#35 call_function (tstate=tstate@entry=0x5555558154b0, pp_stack=pp_stack@entry=0x7fffffffc138, oparg=oparg@entry=3, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:5361
#36 0x00007ffff77b08ef in _PyEval_EvalFrameDefault (tstate=0x5555558154b0,
f=Frame 0x5555558848f0, for file , line 967, in get_code (self=, fullname='test_interpreter', source_path='./test_interpreter.py', source_mtime=1602280047, source_bytes=b'# -*- coding: utf-8 -*-\nfrom widget_module import Widget\n\n\nclass DerivedWidget(Widget):\n def __init__(self, message):\n super(DerivedWidget, self).__init__(message)\n\n def the_answer(self):\n return 42\n', source_hash=None, hash_based=False, check_source=True, bytecode_path='./__pycache__/test_interpreter.cpython-310.pyc', st={'mtime': , 'size': 219}), throwflag=) at ../Python/ceval.c:3784
#37 0x00007ffff769958b in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x5555558848f0, for file , line 967, in get_code (self=, fullname='test_interpreter', source_path='./test_interpreter.py', source_mtime=1602280047, source_bytes=b'# -*- coding: utf-8 -*-\nfrom widget_module import Widget\n\n\nclass DerivedWidget(Widget):\n def __init__(self, message):\n super(DerivedWidget, self).__init__(message)\n\n def the_answer(self):\n return 42\n', source_hash=None, hash_based=False, check_source=True, bytecode_path='./__pycache__/test_interpreter.cpython-310.pyc', st={'mtime': , 'size': 219}), tstate=0x5555558154b0) at ../Include/internal/pycore_ceval.h:40
#38 function_code_fastcall (tstate=0x5555558154b0, co=, args=0x7ffff7ef61e0, args@entry=0x7ffff7ef61d0, nargs=2, globals=) at ../Objects/call.c:330
#39 0x00007ffff7699ee0 in _PyFunction_Vectorcall (func=, stack=0x7ffff7ef61d0, nargsf=, kwnames=) at ../Objects/call.c:367
#40 0x00007ffff77a557a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775810, args=0x7ffff7ef61d0, callable=, tstate=0x5555558154b0)
at ../Include/cpython/abstract.h:114
#41 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775810, args=0x7ffff7ef61d0, callable=) at ../Include/cpython/abstract.h:123
#42 call_function (tstate=tstate@entry=0x5555558154b0, pp_stack=pp_stack@entry=0x7fffffffc368, oparg=oparg@entry=2, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:5361
#43 0x00007ffff77b08ef in _PyEval_EvalFrameDefault (tstate=0x5555558154b0,
f=Frame 0x7ffff7ef6050, for file , line 829, in exec_module (self=, module=), throwflag=) at ../Python/ceval.c:3784
#44 0x00007ffff769958b in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7ffff7ef6050, for file , line 829, in exec_module (self=, module=), tstate=0x5555558154b0) at ../Include/internal/pycore_ceval.h:40
#45 function_code_fastcall (tstate=0x5555558154b0, co=, args=0x7ffff7e5bbe0, args@entry=0x7ffff7e5bbd0, nargs=2, globals=) at ../Objects/call.c:330
#46 0x00007ffff7699ee0 in _PyFunction_Vectorcall (func=, stack=0x7ffff7e5bbd0, nargsf=, kwnames=) at ../Objects/call.c:367
#47 0x00007ffff77a557a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775810, args=0x7ffff7e5bbd0, callable=, tstate=0x5555558154b0)
at ../Include/cpython/abstract.h:114
#48 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775810, args=0x7ffff7e5bbd0, callable=) at ../Include/cpython/abstract.h:123
#49 call_function (tstate=tstate@entry=0x5555558154b0, pp_stack=pp_stack@entry=0x7fffffffc598, oparg=oparg@entry=2, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:5361
#50 0x00007ffff77b08ef in _PyEval_EvalFrameDefault (tstate=0x5555558154b0,
f=Frame 0x7ffff7e5ba50, for file , line 698, in _load_unlocked (spec=, origin='./test_interpreter.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='./__pycache__/test_interpreter.cpython-310.pyc', _initializing=True) at remote 0x7ffff7e8d2d0>, module=), throwflag=) at ../Python/ceval.c:3784
#51 0x00007ffff769958b in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7ffff7e5ba50, for file , line 698, in _load_unlocked (spec=, origin='./test_interpreter.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='./__pycache__/test_interpreter.cpython-310.pyc', _initializing=True) at remote 0x7ffff7e8d2d0>, module=), tstate=0x5555558154b0) at ../Include/internal/pycore_ceval.h:40
#52 function_code_fastcall (tstate=0x5555558154b0, co=, args=0x5555558b9778, args@entry=0x5555558b9770, nargs=1, globals=) at ../Objects/call.c:330
#53 0x00007ffff7699ee0 in _PyFunction_Vectorcall (func=, stack=0x5555558b9770, nargsf=, kwnames=) at ../Objects/call.c:367
#54 0x00007ffff77a557a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x5555558b9770, callable=, tstate=0x5555558154b0)
at ../Include/cpython/abstract.h:114
#55 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x5555558b9770, callable=) at ../Include/cpython/abstract.h:123
--Type for more, q to quit, c to continue without paging--
#56 call_function (tstate=tstate@entry=0x5555558154b0, pp_stack=pp_stack@entry=0x7fffffffc7d0, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:5361
#57 0x00007ffff77b0a30 in _PyEval_EvalFrameDefault (tstate=0x5555558154b0,
f=Frame 0x5555558b95c0, for file , line 1005, in _find_and_load_unlocked (name='test_interpreter', import_=, path=None, parent='', spec=, origin='./test_interpreter.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='./__pycache__/test_interpreter.cpython-310.pyc', _initializing=True) at remote 0x7ffff7e8d2d0>),
throwflag=) at ../Python/ceval.c:3798
#58 0x00007ffff769958b in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x5555558b95c0, for file , line 1005, in _find_and_load_unlocked (name='test_interpreter', import_=, path=None, parent='', spec=, origin='./test_interpreter.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='./__pycache__/test_interpreter.cpython-310.pyc', _initializing=True) at remote 0x7ffff7e8d2d0>),
tstate=0x5555558154b0) at ../Include/internal/pycore_ceval.h:40
#59 function_code_fastcall (tstate=0x5555558154b0, co=, args=0x55555582e190, args@entry=0x55555582e180, nargs=2, globals=) at ../Objects/call.c:330
#60 0x00007ffff7699ee0 in _PyFunction_Vectorcall (func=, stack=0x55555582e180, nargsf=, kwnames=) at ../Objects/call.c:367
#61 0x00007ffff77a557a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775810, args=0x55555582e180, callable=, tstate=0x5555558154b0)
at ../Include/cpython/abstract.h:114
#62 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775810, args=0x55555582e180, callable=) at ../Include/cpython/abstract.h:123
#63 call_function (tstate=tstate@entry=0x5555558154b0, pp_stack=pp_stack@entry=0x7fffffffca00, oparg=oparg@entry=2, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:5361
#64 0x00007ffff77b0a30 in _PyEval_EvalFrameDefault (tstate=0x5555558154b0,
f=Frame 0x55555582dff0, for file , line 1026, in _find_and_load (name='test_interpreter', import_=, module=
Follow-up: Oh joy. Adding these two lines to the start of TEST_CASE("Restart the interpreter") {
makes a difference:
py::finalize_interpreter();
py::initialize_interpreter();
Minimal reproducer:
TEST_CASE("Reproducer") {
py::finalize_interpreter();
py::initialize_interpreter();
py::module_::import("external_module");
}
This looks like fun.
CPython-only reproducer:
#include <Python.h>
int main() {
Py_InitializeEx(1);
Py_Finalize();
Py_InitializeEx(1);
PyRun_SimpleString("class Widget: pass\n"
"class DerivedWidget(Widget):\n"
" def __init__(self):\n"
" super().__init__()\n");
Py_Finalize();
printf("Works\n");
return 0;
}
If anyone's bored, mind trying this out?
Bisected Python to show the error probably got introduced here: python/cpython#20058. I guess I'll make a bug report; this doesn't seem like obvious code to debug...
Reported here: https://bugs.python.org/issue42882 Thanks @bstaletic for testing out!
Should already be fixed by python/cpython#24193 :-) Hopefully we can soon try this out.
Merged into CPython's master. https://www.python.org/dev/peps/pep-0619/ says a new alpha is expected around "2021-02-01". That's pretty long, though. How do we best "fix" CI, until then?
Isn't 3.10-dev
the current 3.10
HEAD? We've never specified that we're running the latest alpha.
Isn't
3.10-dev
the current3.10
HEAD? We've never specified that we're running the latest alpha.
I don't think so. If I read things correctly on https://github.com/actions/setup-python, it's only using releases? (cfr. also https://github.com/actions/setup-python/blob/66319ca9fa6c19287a320c27bd980953e731b827/src/find-python.ts#L157)
3.10-dev is the last pre-release, so 3.10.0a4. Should we disable this check till 3.10.0a5?
I'd be up for that, as long as we can somehow get a reminder that 3.10.0a5 is out?
Alternatively, temporarily switch to nightly builds, so we can catch things earlier, before alpha releases? Can pyenv install the latest master
? But maybe that's not worth it.
On Linux, we can use deadsnakes, though. I think that one has nightly builds pre-built? See #2746.
Can pyenv install the latest master?
Maybe, but it took a while to support 3.9 after the release, so I kind of doubt it. It's also slow.
deadsnakes
Yes, we could do that.
Seen this crop up in my PR and another:
Example PRs:
2768 | https://github.com/pybind/pybind11/pull/2768/checks?check_run_id=1656501003#step:12:36
2772 | https://github.com/pybind/pybind11/pull/2772/checks?check_run_id=1665608739#step:12:35
Guessing it's just a minor thing that'll blow over. I guess we can close this if we don't see it reoccur?
FYI @henryiii @YannickJadoul