3.12 rebase checklist - Githubissues

[x] ae289ce53e Disable the GIL by default.
[ ] edc3cffede Use per-thread refcounts for heap type objects
[x] ~~c734997c4b Implement new interpreter~~
[x] 11de39ee87 Remove "--with-trace-refs" from configure
[x] 59a4843e43 faulthandler: don't dump all threads when running without the GIL
[x] fa42f3bc93 json: make JSON scanner thread safe
[x] 71823e6d9e signalmodule: fix thread-safety issue on macOS
[x] a8ed04a0b7 _struct: fix race condition in cache_struct_converter
[x] 31caeb812b module: intern module name [merged with importlib commit]
[x] d55c608784 sha1: make sha1module thread-safe
[x] 6a9619572e stringio: make stringio thread-safe
[x] e29f51cbb6 semaphore.c: decrease count before release sem_lock
[x] 11de5c8018 importlib: fix data race in imports (PyImport_ImportModuleLevelObject)
[x] 80379ce171 deque: make most functions thread-safe
[x] ~~36d9f3c9aa test_gdb: nogil mode fixes~~
[x] 99ec8d4887 textio: add locks to make textio thread-safe
[x] 945176f2dc bufferedio: add locks to make print() thread-safe
[x] e5e9f2596b clinic: support '@' syntax for recursive mutexes
[x] b10476d76b random: add a recursive mutex to guard random.Random
[x] 46702d3a4c functools: make lru_cache thread-safe
[x] 39dc33ca49 socketmodule.c: use relaxed atomics for global 'defaulttimeout'
[x] 7dafcd2eb9 asdl: use _PyOnceFlag in Python-ast.c
[x] 0f7a4a28bf object.c: fix race when accessing attributes and methods
[x] bf7c7149ce getargs.c: make parser_init thread-safe
[x] e891fff93d asyncio: fix race conditions in enter_task and leave_task
[x] 212f493e36 _threadmodule: thread-safety fixes
[x] eb67b2748a moduleobject: fix data races
[ ] 8ffd4ceab1 ceval: fix data races in ceval.c
[x] ~~4bcc712534 frame: don't use zombieframe when running without the GIL~~
[x] ~~d3503e92a4 accu: don't acquire list lock when appending~~
[x] 7c2790b622 set: make set thread-safe
[x] 17d1edb8c0 list: make list thread-safe
[x] 0d331528ec dict: make dict thread-safe
[x] c2a54b7685 ordereddict: remove C implementation
[x] 90d16587e2 ordereddict: simplify Python implementation
[x] c06dbaad27 queue: make SimpleQueue thread-safe
[x] a33fcf51b7 Stop the world before fork() and Python shutdown
[x] 73af0aecfb gc: implement stop-the-world GC
[x] 9c2c881ed2 threading: remove _tstate_lock from threading.Thread
[x] ~~87364a937a typeobject: remove static cache in resolve_slotdups~~
[x] 2642e687c5 typeobject: make method_cache thread-local
[x] 68bc87fdad _threadmodule: make _thread.lock thread-safe
[x] 66c47e918e codecs.c: fix race condition
[x] 4ed4b0955c unicode: make unicodeobject.c thread-safe
[x] e7a0521ff4 dtoa: make dtoa thread-safe
[x] e390b81964 weakref: make weakrefs thread-safe without the GIL
[x] 7d2cb67605 gc: Traverese mimalloc heaps to find all objects.
[x] 4778bf9f39 gc: make the garbage collector non-generational
[x] d8326b0620 obmalloc: use separate mimalloc heaps in obmalloc.c
[x] d059e29758 mimalloc: tag pages with qsbr counter
[x] edc7b51d49 mimalloc: don't fill in first N bytes with debug values
[x] 383f593135 mimalloc: split thread exiting from abandoning heaps
[x] d340a94863 mimalloc: changes to support separate heaps
[x] 9a7f2267df mimalloc: minimal changes for use in Python
[x] c87a2a6ca2 Add mimalloc v1.7.5
[x] 79d522f093 Create io.BytesIO() in destructor of test_io.py
[x] ~~cbbf4d38c0 Disable most free lists~~
[x] 6f4617626b Add safe memory reclamation scheme based on FreeBSD's GUS
[x] ca2a8fdd8f Implement biased reference counting
[x] 2205f705fd Move _Py_RefTotal to per-thread state
[x] ~~9c5049d351 Call function when incref/decref~~
[x] ~~c40dda7f82 Add ob_tid to every object~~
[x] 50c13031e7 Make global objects immortal (not reference counted)
[x] 7884d98939 multiprocessing: use CriticalLock in a few places
[x] 350b0cdfb1 Add _thread.CriticalLock which prevents the current thread from parking
[x] a03f4f73ee ceval: move eval_breaker to per-thread state
[x] 3a98232c45 Enable/disable the GIL at runtime
[x] d1c164e8c7 Change "SOABI" and sys.implementation.name and define Py_NOGIL
[x] 0951ec4e55 critical_section: helpers for fine-grained locking
[x] c3272efb8b parking_lot: add mutexes and one-time notifications
[x] 9bb0733c89 pystate: keep track of attached vs. detached state
[x] d1aee8abd7 Skip subinterpreter tests that run on "incorrect" thread
[x] 0b8a36d4f8 Make PyThreadState_GET thread-local
[x] 9c38274cd9 pyport: add new macros
[x] bd068acb8a Add pyatomic.h

More recent changes (not all need to be rebased):

[x] 4de17ac5e9 pyatomic: fix warning on MSVC
[ ] e7d66cf129 pip: update modified bundled pip
[ ] 969ec6b2a0 Fix platform.python_compiler()
[x] 5a7b97eab0 atomic: add bulitin (GCC) atomics
[x] 22480e770f pyatomic: error when there is atomic implementation
[ ] 0085e2ce3c Overwite INCLUDEPY and CONFINCLUDEPY
[ ] 258cab1c76 Update pythonba.vcxproj from upstream CPython
[ ] ee13015e1a Update windows.yml
[ ] f0e7b22bf7 Don't compile ceval.s with ceval.o
[ ] 52e8523021 Add GitHub actions to build Windows and macOS distributions
[ ] 965e9670fc ceval: Mark functions as always-inlined
[ ] 4b05376b6d Fix gdb traceback for Python generators
[ ] f7e45d6bfb Update publish-docker.yml
[ ] 1761801084 Merge pull request colesbury/nogil#85 from lesteve/docker-build-push
[ ] f34229def1 Fix docker tag
[ ] ca01d13d6b Add github workflow to build and publish docker image
[ ] 9bb1e2edd1 Pin jinja2 version in Doc/make.bat
[ ] 44a22d0091 Tools/msi: add buildnogil.bat
[ ] 49ce508398 bpo-47138: Fix documentation build by pinning Jinja version to 3.0.3 (GH-32153)
[ ] 0ad343d10a Make small ints immortal
[ ] 55d895ff81 PCBuild: add missing file 'bitmap.c'
[ ] 2dcbadc25d frame: fix thread-safety issue when accessing f_lineno
[x] 4b358237b4 list: make list_repr thread-safe
[ ] f3b6e462dc code: improve hash function
[ ] e664310886 Fix makeopcodetargets.py and generated files after recent changes
[ ] 8ad4f8aea0 dis: make dis.opmap values be ints instead of Bytecode objects
[ ] fc44b03fa2 dis: prefer string/const names as "argval"
[ ] d0dfef6352 code: make co_names an alias for co_consts
[ ] 411e8494ce docker: fix key issue with NVIDIA CUDA image
[ ] 463fc64546 code: rename co_framesize to co_stacksize
[ ] f33857972b pystate: re-init stoptheworld_mutex after-fork
[ ] 0126b8947b code: use empty tuple when missing some attributes
[ ] a6976eb0cb tuple: expose _Py_EmptyTuple to limited API
[ ] 9678b94533 Move critical_section to the limited API
[ ] b1689ff536 ceval: clean up vm_trace
[ ] 74e57b70e8 ceval: fix detection of Python function calls
[ ] e785b9c991 signal: ensure signal handler function is retained across call
[ ] 6c3895ff21 ceval: add missing cororoutine return trace events
[ ] 0453fa68bc ceval: fix tracing of C method descriptors when called with varargs
[ ] ca53bb4573 Update and rename README.rst to README.md
[ ] 5c35dc0df4 Add "nogil" identifier to Py_GetVersion()
[ ] bb728e35a9 Add Py_RefCnt function for use from PyO3 fork
[ ] 87c61c34a1 funcobject: missing Py_INCREF when replacing closure
[x] df2ffbdfa3 mimalloc: create Objects/mimalloc build directory during configure
[ ] f9dbb92afa code: fix bug in code_richcompare
[ ] df289e6bbf list: fix thread-safety issues in list_ass_slice
[ ] 9d8a660c2b list: fix thread-safety issue with list slicing
[ ] 7d1be8bac8 remove duplicate typedefs
[ ] c3894a5ead pip: use modified bundled pip
[ ] 895c1244da Add docker files for building nogil Python
[ ] 0af745b7dd Update README.rst for nogil

Deferred tasks:

[x] check list capacity in lock-less fetch
[x] Ensure that all interned strings are immortalized
[x] list: mark arrays freed during some _list_clear calls
[x] ~~fix _mi_page_use_qsbr on Windows (move to _PyMem ?)~~
[x] Lots of dict thread-safety issues, including using ma_version_tag in lockless fetch
[x] Python GC header shrinkage
[ ] Make more types immortal
[x] dict version tag and keys tag thread-safety
[ ] list: validate safety of sort
[x] Use critical sections in deque and clinic
[ ] Call _Py_qsbr_quiescent_state occasionally
[ ] cell & maybe weakrefs

Dictionary TODOs:

[x] Don't reference count empty_keys_struct
[x] Use deferred reference counting for class keys (CACHED_KEYS)
[x] Thread-safe DICT_NEXT_VERSION()
[x] Thread-safe dk_version / next_keys_version (dict keys version)
[x] dk_mutex lock should not detach thread when it blocks (otherwise dictionary might change underneath us).
[x] qsbr for resized keys
[ ] PyDict_Contains
[ ] dictionary iterators
[ ] Audit functions
[ ] test.test_multiprocessing_spawn.WithProcessesTestFinalize.test_thread_safety: RuntimeError: dictionary keys changed during iteration (File "/raid/sgross/nogil/Lib/multiprocessing/util.py", line 294, in _run_finalizers) registry = list(_finalizer_registry)

TSAN races to fix:

[x] Modification of keys->dk_usable in init_inline_values with non-atomic read in insert_into_dictkeys
[x] ~~False positive in threadmodule.c (self->locked). sem_clockwait is not instrumented (I think). Is it still being called under TSAN?~~
[x] PyMem_RawFree(interp) vs. _PyEval_ReleaseLock (tstate->interp->runtime->ceval) called from _PyThreadState_DeleteCurrent. (Fix todo: check gil->enabled before dereferencing tstate->interp)
[x] use_tracing (via PyEval_SetProfileAllThreads and _PyEval_SetProfile): https://github.com/colesbury/nogil-3.12/blob/78050fa422e93d3eb90d7be00a7600d08381d889/Include/internal/pycore_pystate.h#L206 and https://github.com/colesbury/nogil-3.12/blob/78050fa422e93d3eb90d7be00a7600d08381d889/Python/generated_cases.c.h#L714 (Stop the world? Or concurrently enable tracing?)
[x] _PyRuntime.stop_the_world (Should it be set after the world is stopped or at request?) https://github.com/colesbury/nogil-3.12/blob/78050fa422e93d3eb90d7be00a7600d08381d889/Python/pystate.c#L317 and https://github.com/colesbury/nogil-3.12/blob/78050fa422e93d3eb90d7be00a7600d08381d889/Objects/object.c#L2601
[x] free_threadstate e.g. https://github.com/colesbury/nogil-3.12/blob/78050fa422e93d3eb90d7be00a7600d08381d889/Python/pystate.c#L984 and https://github.com/colesbury/nogil-3.12/blob/78050fa422e93d3eb90d7be00a7600d08381d889/Python/ceval_gil.c#L228 (GIL is re-enabled in subtest, but problem exists generally because main thread deletes thread states while those threads are still running. Now that eval_breaker is in threadstate, this is problematic. Maybe refcount thread states is easiest fix.)
[x] threadmodule.c: threading.Lock() (i.e., _thread.LockType) supports acquiring/releasing from different threads. Not currently thread-safe without the GIL from different threads.
[x] interp->slice_cache
[x] Modules/_multiprocessing/semaphore.c read of count/tid

Remaining nogil bugs:

[x] test_import (test_concurrency):


python: Python/generated_cases.c.h:1336: _PyEval_EvalFrameDefault: Assertion `cframe.use_tracing == 0' failed.
Fatal Python error: Aborted

Stack (most recent call first): File "/raid/sgross/nogil/Lib/threading.py", line 329 in wait File "/raid/sgross/nogil/Lib/threading.py", line 636 in wait File "/raid/sgross/nogil/Lib/test/test_import/init.py", line 462 in run File "/raid/sgross/nogil/Lib/threading.py", line 965 in run File "/raid/sgross/nogil/Lib/threading.py", line 1014 in _bootstrap_inner File "/raid/sgross/nogil/Lib/threading.py", line 985 in _bootstrap


- [x] test_sys (test_current_exceptions)

Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2) Warning -- Dangling thread: <Thread(Thread-1 (f123), started 139718199457536)> Warning -- Dangling thread: <_MainThread(MainThread, started 139718204011136)> Warning -- threading._dangling was modified by test_sys Warning -- Before: {<weakref at 0x4804cf0a460; to '_MainThread' at 0x4804c383be0>} Warning -- After: {<weakref at 0x4804cf08f60; to 'Thread' at 0x4804d1c0a20>, <weakref at 0x4804cf0bff0; to '_MainThread' at 0x4804c383be0>} test test_sys failed -- Traceback (most recent call last): File "/raid/sgross/nogil/Lib/test/support/threading_helper.py", line 63, in decorator return func(*args) ^^^^^^^^^^^ File "/raid/sgross/nogil/Lib/test/test_sys.py", line 520, in test_current_exceptions stack = traceback.extract_stack(exc_tb.tb_frame) ^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'tb_frame'

{"name": "test_sys", "duration_sec": 4.2834788602776825, "xml_data": null, "errors": [["test_current_exceptions (test.test_sys.SysModuleTest.test_current_exceptions)", "Traceback (most recent call last):\ n File \"/raid/sgross/nogil/Lib/test/support/threading_helper.py\", line 63, in decorator\n return func(*args)\n ^^^^^^^^^^^\n File \"/raid/sgross/nogil/Lib/test/test_sys.py\", line 520, in test_current_exceptions\n stack = traceback.extract_stack(exc_tb.tb_frame)\n ^^^^^^^^^^^^^^^\nAttributeError: 'NoneType' object has no attribute 'tb_frame'\n"]], "fa ilures": [], "__test_result__": "Failed"} Timeout (0:02:00)! Thread 0x00007f12ada5b700 (most recent call first): File "/raid/sgross/nogil/Lib/threading.py", line 340 in wait File "/raid/sgross/nogil/Lib/threading.py", line 636 in wait File "/raid/sgross/nogil/Lib/test/test_sys.py", line 493 in g456 File "/raid/sgross/nogil/Lib/test/test_sys.py", line 484 in f123 File "/raid/sgross/nogil/Lib/threading.py", line 965 in run File "/raid/sgross/nogil/Lib/threading.py", line 1014 in _bootstrap_inner File "/raid/sgross/nogil/Lib/threading.py", line 985 in _bootstrap

Thread 0x00007f12adeb3280 (most recent call first): File "/raid/sgross/nogil/Lib/threading.py", line 1478 in _shutdown

colesbury / nogil-3.12

3.12 rebase checklist #2