search

aio-libs / aiohttp

Asynchronous HTTP client/server framework for asyncio and Python
https://docs.aiohttp.org
Other
14.98k stars 2k forks source link

[3.9] test_max_age segfaults #7851

Closed mgorny closed 9 months ago

mgorny commented 10 months ago

Describe the bug

When running the test suite, tests/test_cookiejar.py::TestCookieJarSafe::test_max_age segfaults. I've been able to hit it both with 3.9.0 sdist and with git checkout as of 6f7667336b92dbf09a6a283c2496cdd64ad56a62.

It's somehow related to time-machine but all its tests pass here, so I suspect aiohttp may be doing something to trigger it.

To Reproduce

  1. python3.11 -m venv .venv
  2. make
  3. python -m pytest -s tests/test_cookiejar.py::TestCookieJarSafe::test_max_age

Expected behavior

Tests passing.

Logs/tracebacks

========================================================= test session starts =========================================================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.2.0 -- /tmp/aiohttp/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/aiohttp
configfile: setup.cfg
plugins: time-machine-2.13.0, mock-3.12.0, cov-4.1.0
collected 1 item                                                                                                                      

tests/test_cookiejar.py::TestCookieJarSafe::test_max_age Fatal Python error: Segmentation fault

Current thread 0x00007f3941391740 (most recent call first):
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/time_machine/__init__.py", line 405 in time
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/time_machine/__init__.py", line 129 in extract_timestamp_tzname
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/time_machine/__init__.py", line 229 in __init__
  File "/tmp/aiohttp/tests/test_cookiejar.py", line 430 in timed_request
  File "/tmp/aiohttp/tests/test_cookiejar.py", line 650 in test_max_age
  File "/usr/lib/python3.11/unittest/case.py", line 579 in _callTestMethod
  File "/usr/lib/python3.11/unittest/case.py", line 623 in run
  File "/usr/lib/python3.11/unittest/case.py", line 678 in __call__
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/unittest.py", line 333 in runtest
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/tmp/aiohttp/.venv/lib/python3.11/site-packages/pytest/__main__.py", line 5 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: _time_machine, multidict._multidict, yarl._quoting_c, _brotli, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, _cffi_backend, frozenlist._frozenlist, uvloop.loop (total: 11)
Segmentation fault (core dumped)

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f3940aa2e2f in __pthread_kill_internal (signo=11, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f3940a52cc2 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
#3  0x00007f3940f7ed3e in faulthandler_fatal_error (signum=11) at ./Modules/faulthandler.c:384
#4  <signal handler called>
#5  0x0000000000000000 in ?? ()
#6  0x00007f393f10da1c in _time_machine_original_time (module=<module at remote 0x7f393f1ef510>, args=0x0) at src/_time_machine.c:336
#7  0x00007f3940df1ff7 in cfunction_vectorcall_NOARGS (
    func=<built-in method original_time of module object at remote 0x7f393f1ef510>, args=0x7f394147d870, nargsf=0, kwnames=0x0)
    at Objects/methodobject.c:486
#8  0x00007f3940d84363 in _PyObject_VectorcallTstate (tstate=0x7f39412a9218 <_PyRuntime+166328>, 
    callable=<built-in method original_time of module object at remote 0x7f393f1ef510>, args=0x7f394147d870, nargsf=0, kwnames=0x0)
    at ./Include/internal/pycore_call.h:92
#9  0x00007f3940d850e2 in PyObject_Vectorcall (callable=<built-in method original_time of module object at remote 0x7f393f1ef510>, 
    args=0x7f394147d870, nargsf=0, kwnames=0x0) at Objects/call.c:299
#10 0x00007f3940ee3c64 in trace_call_function (tstate=0x7f39412a9218 <_PyRuntime+166328>, 
    func=<built-in method original_time of module object at remote 0x7f393f1ef510>, args=0x7f394147d870, nargs=0, kwnames=0x0)
    at Python/ceval.c:7279
#11 0x00007f3940ed814a in _PyEval_EvalFrameDefault (tstate=0x7f39412a9218 <_PyRuntime+166328>, frame=0x7f394147d810, throwflag=0)
    at Python/ceval.c:4755
#12 0x00007f3940ebfa5a in _PyEval_EvalFrame (tstate=0x7f39412a9218 <_PyRuntime+166328>, frame=0x7f394147d6f8, throwflag=0)
    at ./Include/internal/pycore_ceval.h:73
#13 0x00007f3940ee1162 in _PyEval_Vector (tstate=0x7f39412a9218 <_PyRuntime+166328>, func=0x7f393eeddc60, locals=0x0, 
    args=0x7f393a9697b8, argcount=2, kwnames=('tick',)) at Python/ceval.c:6425
#14 0x00007f3940d85531 in _PyFunction_Vectorcall (func=<function at remote 0x7f393eeddc60>, stack=0x7f393a9697b8, 
    nargsf=9223372036854775810, kwnames=('tick',)) at Objects/call.c:393
#15 0x00007f3940d849cf in _PyObject_FastCallDictTstate (tstate=0x7f39412a9218 <_PyRuntime+166328>, 
    callable=<function at remote 0x7f393eeddc60>, args=0x7ffd7a46b960, nargsf=2, kwargs={'tick': False}) at Objects/call.c:152
#16 0x00007f3940d85899 in _PyObject_Call_Prepend (tstate=0x7f39412a9218 <_PyRuntime+166328>, 
    callable=<function at remote 0x7f393eeddc60>, obj=<travel() at remote 0x7f393a98e850>, 
    args=(<datetime.timedelta at remote 0x7f393a96e790>,), kwargs={'tick': False}) at Objects/call.c:482
#17 0x00007f3940e2634b in slot_tp_init (self=<travel() at remote 0x7f393a98e850>, 
    args=(<datetime.timedelta at remote 0x7f393a96e790>,), kwds={'tick': False}) at Objects/typeobject.c:7854
#18 0x00007f3940e1302f in type_call (type=0x56107db34d40, args=(<datetime.timedelta at remote 0x7f393a96e790>,), kwds={'tick': False})
    at Objects/typeobject.c:1103
#19 0x00007f3940d84cc9 in _PyObject_MakeTpCall (tstate=0x7f39412a9218 <_PyRuntime+166328>, callable=<type at remote 0x56107db34d40>, 
    args=0x7f394147d6d8, nargs=1, keywords=('tick',)) at Objects/call.c:214
#20 0x00007f3940d84347 in _PyObject_VectorcallTstate (tstate=0x7f39412a9218 <_PyRuntime+166328>, 
    callable=<type at remote 0x56107db34d40>, args=0x7f394147d6d8, nargsf=9223372036854775809, kwnames=('tick',))
    at ./Include/internal/pycore_call.h:90
#21 0x00007f3940d850e2 in PyObject_Vectorcall (callable=<type at remote 0x56107db34d40>, args=0x7f394147d6d8, 
    nargsf=9223372036854775809, kwnames=('tick',)) at Objects/call.c:299
#22 0x00007f3940ee3f05 in trace_call_function (tstate=0x7f39412a9218 <_PyRuntime+166328>, func=<type at remote 0x56107db34d40>, 
    args=0x7f394147d6d8, nargs=1, kwnames=('tick',)) at Python/ceval.c:7301
#23 0x00007f3940ed814a in _PyEval_EvalFrameDefault (tstate=0x7f39412a9218 <_PyRuntime+166328>, frame=0x7f394147d658, throwflag=0)
    at Python/ceval.c:4755
#24 0x00007f3940ebfa5a in _PyEval_EvalFrame (tstate=0x7f39412a9218 <_PyRuntime+166328>, frame=0x7f394147d498, throwflag=0)
    at ./Include/internal/pycore_ceval.h:73
#25 0x00007f3940ee1162 in _PyEval_Vector (tstate=0x7f39412a9218 <_PyRuntime+166328>, func=0x7f393f44efc0, locals=0x0, 
    args=0x7f39409a30f0, argcount=1, kwnames=('result',)) at Python/ceval.c:6425
#26 0x00007f3940d85531 in _PyFunction_Vectorcall (func=<function at remote 0x7f393f44efc0>, stack=0x7f39409a30f0, nargsf=1, 
    kwnames=('result',)) at Objects/call.c:393
#27 0x00007f3940d8893f in _PyObject_VectorcallTstate (tstate=0x7f39412a9218 <_PyRuntime+166328>, 
    callable=<function at remote 0x7f393f44efc0>, args=0x7f39409a30f0, nargsf=1, kwnames=('result',))
    at ./Include/internal/pycore_call.h:92
#28 0x00007f3940d89004 in method_vectorcall (method=<method at remote 0x7f393a970740>, args=0x7f39409a30f8, 
    nargsf=9223372036854775808, kwnames=('result',)) at Objects/classobject.c:59
#29 0x00007f3940d84f0a in _PyVectorcall_Call (tstate=0x7f39412a9218 <_PyRuntime+166328>, func=0x7f3940d88ef7 <method_vectorcall>, 
    callable=<method at remote 0x7f393a970740>, tuple=(), kwargs={'result': <TestCaseFunction at remote 0x7f393a975cc0>})
    at Objects/call.c:257
#30 0x00007f3940d85233 in _PyObject_Call (tstate=0x7f39412a9218 <_PyRuntime+166328>, callable=<method at remote 0x7f393a970740>, 
    args=(), kwargs={'result': <TestCaseFunction at remote 0x7f393a975cc0>}) at Objects/call.c:328
#31 0x00007f3940d8532c in PyObject_Call (callable=<method at remote 0x7f393a970740>, args=(), 
    kwargs={'result': <TestCaseFunction at remote 0x7f393a975cc0>}) at Objects/call.c:355
#32 0x00007f3940ee4445 in do_call_core (tstate=0x7f39412a9218 <_PyRuntime+166328>, func=<method at remote 0x7f393a970740>, 
    callargs=(), kwdict={'result': <TestCaseFunction at remote 0x7f393a975cc0>}, use_tracing=255) at Python/ceval.c:7343
#33 0x00007f3940edc36e in _PyEval_EvalFrameDefault (tstate=0x7f39412a9218 <_PyRuntime+166328>, frame=0x7f394147d410, throwflag=0)
    at Python/ceval.c:5367
#34 0x00007f3940ebfa5a in _PyEval_EvalFrame (tstate=0x7f39412a9218 <_PyRuntime+166328>, frame=0x7f394147d410, throwflag=0)
    at ./Include/internal/pycore_ceval.h:73
[...]
(gdb) up
#6  0x00007f393f10da1c in _time_machine_original_time (module=<module at remote 0x7f393f1ef510>, args=0x0) at src/_time_machine.c:336
336     PyObject* result = state->original_time(time_module, args);
(gdb) p *state
$2 = {original_now = 0x0, original_utcnow = 0x0, original_clock_gettime = 0x0, original_clock_gettime_ns = 0x0, 
  original_gmtime = 0x0, original_localtime = 0x0, original_monotonic = 0x0, original_monotonic_ns = 0x0, original_strftime = 0x0, 
  original_time = 0x0, original_time_ns = 0x0}

Python Version

$ python --version
Python 3.11.6

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 4.0.0a2.dev0
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: 
Author-email: 
License: Apache 2
Location: /tmp/aiohttp/.venv/lib/python3.11/site-packages
Editable project location: /tmp/aiohttp
Requires: aiosignal, frozenlist, multidict, yarl
Required-by:

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /tmp/aiohttp/.venv/lib/python3.11/site-packages
Requires: 
Required-by: aiohttp, yarl

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.9.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: /tmp/aiohttp/.venv/lib/python3.11/site-packages
Requires: idna, multidict
Required-by: aiohttp

OS

Gentoo Linux amd64

Related component

Server, Client

Additional context

No response

Code of Conduct

Dreamsorcerer commented 10 months ago

Hmm, we know that time-machine is only expected to work on cpython, as it uses some internal hooks or something. Is it possible that your version is patched in such a way that it becomes incompatible?

mgorny commented 10 months ago

Hmm, we know that time-machine is only expected to work on cpython, as it uses some internal hooks or something. Is it possible that your version is patched in such a way that it becomes incompatible?

No, we're not applying any such patches. Also, time-machine's tests all pass and I haven't seen any other package crash with it (requests-cache, tox and virtualenv).

Dreamsorcerer commented 10 months ago

Hmm, the traceback appears that it is failing inside time-machine though, so maybe worth opening an issue there?

mgorny commented 10 months ago

Sure, I can do that.

mgorny commented 10 months ago

Ok, it seems that time-machine crashes when passing a timedelta object to travel(), even though docs suggest that it's a valid use. I'm not going to insist but I was able to work around the problem by using datetime.datetime.now() + ... instead.

Dreamsorcerer commented 10 months ago

I guess we can wait a few days and see if they fix it in time-machine.

mgorny commented 9 months ago

I'm afraid there doesn't seem to be any activity on the time-machine end. Could you apply the workaround, please?

I really do miss the reliable simplicity of freezegun…

Dreamsorcerer commented 9 months ago

Yeah, we looked at time-machine because the last comitter to freezegun said they had given up maintaining freezegun and that time-machine was a better alternative. But, it's starting to look like it has more problems and is just as unmaintained...

Dreamsorcerer commented 9 months ago

Although, actually, there are commits to the beginning of November, maybe they've just taken a long holiday and will be back in January...?

mgorny commented 9 months ago

Yeah, let's hope situation improves. That said, there's very little chance for PyPy support, and I really wish time-machine featured pure Python fallback that would both improve portability and give us an easy way out in case of segfaults.

Dreamsorcerer commented 9 months ago

And, just as time-machine goes silent, freezegun wakes up...

Will see if there's a new release shortly, and then probably revert the commit back to freezegun in that case.