Open colesbury opened 3 months ago
What concrete parts of the stdlib or tests have this race condition?
There were some issues in multiprocessing, I tried to avoid introducing such kind of race condition.
@serhiy-storchaka I don't think the stdlib has internal races on file descriptors. We have some in tests, but it may take me a bit to find them because we still have other race conditions reported by TSan, and it's easy to get them mixed up.
test_socket.testClose
- race between recv()
and connect()
reported by TSan. This seems benign to me -- unlike my original example, neither call creates or closes file descriptors -- but I'm not entirely sure.
Python allows multiple threads to concurrently access file descriptors through files and sockets. Race conditions at the Python level can lead to unexpected behavior, even with the global interpreter lock.
Thread sanitizer reports these races in some of our tests that exercise this behavior. We cannot fix these potential race conditions without introducing potential deadlocks.
For example, consider:
If you are particularly unlucky, then the file descriptor for
secrets
may be closed bythread2
and reused as the file descriptor forlog
just beforethread1
writes to it. In other words,thread1
may write "a secret" to a completely different file or socket than it intended.This can happen, even with the GIL, because the GIL is released around
write()
andclose()
. In other words, the following can happen:secrets.write()
call releases the GIL, but before it actually calls the Cwrite()
function on the file descriptor....secrets.close()
call closes the file descriptoropen("log.txt", "w")
re-uses the same file descriptor numbersecrets.write()
call nowwrite()
on the wrong file descriptorNote that we must release the GIL before calling
write()
orclose()
because these functions potentially block.