python / cpython

The Python programming language
https://www.python.org
Other
63.38k stars 30.35k forks source link

Make TSAN tests pass with the GIL disabled in free-threaded builds #117657

Open mpage opened 7 months ago

mpage commented 7 months ago

Feature or enhancement

We need to make the TSAN tests pass with the GIL disabled before we can disable the GIL by default in free-threaded builds.

This should proceed in two phases:

  1. Add suppressions for existing TSAN warnings.
  2. Triage, fix, and remove the suppressions for the warnings enumerated in (1).

How to run the TSAN tests

  1. Build with TSAN: env CC=clang CXX=clang++ ./configure --disable-gil --with-thread-sanitizer --with-pydebug && make -j
  2. Run tests: env TSAN_OPTIONS=suppressions=<repo_root>/Tools/tsan/suppressions_free_threading.txt ./python -m test --tsan -j4

Working on a race

  1. Verify that the TSAN tests are passing using the steps from above.
  2. Pick a suppression from the section below and assign it to yourself (edit it and add your username next to it). Some of them may be related. If you find that other suppressions are related to the race you're working on please assign them to yourself or contact their owner if they're already assigned.
  3. Delete the suppression from <repo_root>/Tools/tsan/suppressions_free_threading.txt, run the TSAN tests, and verify that the race is reported by TSAN. You may need to comment out unrelated functions (notably, _PyEval_EvalFrameDefault) in order to reproduce the race.
  4. Fix the race.

Suppressions

### Tasks
- [ ] PyInterpreterState_ThreadHead
- [ ] PyThreadState_Clear
- [ ] PyThreadState_Next
- [ ] PyUnstable_InterpreterFrame_GetLine
- [ ] _PyFrame_GetCode
- [ ] _PyFrame_Initialize
- [ ] _PyObject_TryGetInstanceAttribute
- [ ] _Py_slot_tp_getattr_hook
- [x] _multiprocessing_SemLock_acquire_impl
- [x] _multiprocessing_SemLock_release_impl
- [ ] add_threadstate
- [ ] assign_version_tag
- [ ] dump_traceback
- [ ] fatal_error
- [ ] free_threadstate
- [ ] list_get_item_ref
- [ ] new_reference
- [ ] pthread_create
- [ ] set_default_allocator_unlocked
- [ ] tstate_delete_common
- [ ] tstate_is_freed
- [ ] type_modified_unlocked
- [ ] write_thread_id

Linked PRs

SonicField commented 6 months ago

I will look at initialize_new_array

SonicField commented 6 months ago

I'll pick up insertdict and lookup_tp_dict now I have initialize_new_array done. I'll bundle all three into a pull request. Sounds reasonable?

colesbury commented 6 months ago

@SonicField - great! Would you please mark the initialize_new_array as "ready"?