Improving test tools and strategies for multi-threading

Quansight-Labs / free-threaded-compatibility

A central repository to keep track of the status of work on and support for free-threaded CPython (see PEP 703), with a focus on the scientific and ML/AI ecosystem

https://py-free-threading.github.io/

MIT License

150 stars 21 forks source link

Improving test tools and strategies for multi-threading #56

Open rgommers opened 2 months ago

rgommers commented 2 months ago

The suggested plan of attack and documented test strategies on https://py-free-threading.github.io are up-to-date given what we know and the tools we have available. However, the testing strategies for uncovering thread-safety issues are falling short of what we need. Improving this should be a priority.

Ideas that need implementing or protyping:

[ ] create a decorator to easily run a pytest test case in parallel (can be based on the test patterns for threading and concurrent.futures in this doc section, and look something like the run_in_parallel from scikit-image)
[ ] a pytest plugin to all tests in a full test suite in parallel (e.g. using the decorator from the task above)
[ ] use ThreadSanitizer on a Python package (it has proven useful for CPython)
- [ ] set up for NumPy and figure out how useful it is (what does it catch, what's the false positive rate)
- [ ] document how to do this with Meson and CMake

ngoldbaum commented 2 months ago

For TSAN, I found some meson configuration in the QEMU repo that might be something to start with. It looks like all that's needed is making sure -fsanitize=thread gets into CFLAGS and LDFLAGS.

ngoldbaum commented 2 months ago

Oh never mind, meson has built-in support for it via b_sanitize. For some reason I thought there wasn't support.

h-vetinari commented 2 months ago

https://py-free-threading.github.io looks like a cool resource! Are you planning to announce it at some point?

ngoldbaum commented 2 months ago

We did, via a blog post. Is there a spot you would have more easily heard the announcement?

ngoldbaum commented 2 months ago

I spent a little time looking at setting up TSAN for NumPy. Unfortunately it's not quite as easy to get working as ASAN. I got it working on my Mac, but because you need to set DYLD_INSERT_LIBRARIES and system integrity protection means that subprocesses don't inherit that environment variable, many tests fail because of that. Eventually I hit a seg fault in test_dtype.py::TestMonsterType::test_dict_recursion which I suspect is causing a stack overflow under TSAN. I'll have to look closer.

I tried also on my x86 linux laptop and unfortunately in Linux TSAN has issues with ASLR on newer kernels: https://github.com/google/sanitizers/issues/1716#issuecomment-2010399341

I might be able to get this working with clang 18 on Linux, but that might be annoying to get working on CI since it looks like that version hasn't been packaged yet for Ubuntu.

h-vetinari commented 2 months ago

We did, via a blog post. Is there a spot you would have more easily heard the announcement?

I think an announcement on discuss.python.org would make sense...? 🤔

h-vetinari commented 2 months ago

I might be able to get this working with clang 18 on Linux, but that might be annoying to get working on CI since it looks like that version hasn't been packaged yet for Ubuntu.

It's packaged (also for linux) in conda-forge. ;)

conda install clang_linux-64=18

h-vetinari commented 2 months ago

Or plain clang if you want the bare binaries without any default flags etc. (which you can of course still override after the environment has been activated)