Add support for thread sanitizer (TSAN) via `--with-thread-sanitizer`

[colesbury]

Feature or enhancement

GCC and Clang provide ThreadSanitizer, a tool that detects data races. We should add support for thread sanitizer to CPython. Note that Python already supports the memory and address sanitizers.

• Add --with-thread-sanitizer as a configure option
• Define _Py_THREAD_SANITIZER if thread-sanitizer is enabled (see, e.g., _Py_ADDRESS_SANITIZER in pyport.h)
• Move the definition of _Py_NO_SANITIZE_THREAD from obmalloc.c to a place that's more widely available (like pyport.h). We're going to need this in a few places

Eventually, it would be helpful to have a continuous build for the combination of --disable-gil --with-thread-sanitizer. Note that we probably won't want to run all the tests. ThreadSanitizer is slow and also not very useful for single-threaded tests. We should collect a subset of our tests that use threading for the ThreadSanitizer continuous build.

Linked PRs

• gh-112648
• gh-113232
• gh-116555
• gh-116558
• gh-116601
• gh-116872
• gh-116896
• gh-116898
• gh-116911
• gh-116924
• gh-116926
• gh-116929
• gh-116953
• gh-117702
• gh-117713
• gh-123833

[samety]

Hey, I would like to pick this up. As a first step tried copying the struct for ASAN to TSAN Here is a draft PR: https://github.com/python/cpython/pull/112648

Happy to get some initial feedback on it and then add remaining things like _Py_THREAD_SANITIZER etc.

[samety]

Added --with-thread-sanitizer as a configure option in pr 112648.

Still looking for a good way to enable TSAN for the continuous build, I will make another PR when I find one.

[pitrou]

Do we want to backport these at some point?

[corona10]

~@pitrou We can go for the default build. I will submit the PR soon~

[corona10]

Ah but the problem is —tsan flag..

[corona10]

If we want to backport the TSAN test, the following PRs should be backported, but this is not a bug or security fix. We may need some discussion.

• https://github.com/python/cpython/pull/112648
• https://github.com/python/cpython/pull/116601
• https://github.com/python/cpython/pull/116872

[pitrou]

Right. Perhaps ask on Discourse?

[corona10]

Right. Perhaps ask on Discourse?

Good, do you want to write it? (I can write it but, since you proposed the suggestion first, I am asking)

[pitrou]

Ok, I posted https://discuss.python.org/t/backport-thread-sanitizer-infrastructure/48704

[corona10]

@pitrou cc @colesbury By the way, once the backport patches are completed, let's close this issue. Expanding TSAN test can be tracked from the new issue. (I prefer to cut off the issues as the checkpoint)

[pitrou]

Ok, closing as done now.

[comicfans]

hello, possible to backport this to 3.10 too ?

[picnixz]

Unfortunately, 3.8 to 3.11 are security-only branches. As such, they would not accept such backports. The 3.11 backport was not merged since it was close to the last bugfix deadline.