python-trio / trio

Trio – a friendly Python library for async concurrency and I/O
https://trio.readthedocs.io
Other
6.09k stars 331 forks source link

Better validate serve_listeners handling of weird edge cases #492

Open njsmith opened 6 years ago

njsmith commented 6 years ago

Looking at https://github.com/twisted/twisted/pull/996/ led me to investigate more the difference between EMFILE and WSAEMFILE... the latter is the special "winsockets" version of EMFILE, and apparently they are actually different:

In [2]: errno.EMFILE
Out[2]: 24

In [3]: errno.WSAEMFILE
Out[3]: 10024

And right now serve_listeners has special handling for EMFILE, but not WSAEMFILE... so it's probably broken.

We should:

njsmith commented 6 years ago

On further investigation, this is not actually a real issue, because Windows doesn't actually have any way to limit the number of socket handles you allocate; if you do a while True: sockets.append(socket.socket()) then it doesn't return any kind of error, it just eventually locks up the whole system because no processes can get any handles.

Paste of some discussion on #twisted with @markrwilliams:

``` njs: also, at some point i'd like to talk to you about WSAEMFILE on windows though i can probably edit the trio issue you linked to that twisted PR err, comment on runciter: sure runciter: I don't really know anything about it yet though :-) njs: https://msdn.microsoft.com/en-us/library/windows/desktop/ms740522(v=vs.85).aspx njs: that's a citation, sort of, for why WSAEMFILE is different from EMFILE - sockets are generally not files as a consequence you can call os.dup(0) in a loop until you hit EMFILE, and then immediately allocate a socket calling socket.socket() in a loop hits some kind of limit, but it's not WSAEMFILE does os.dup() even touch handles at all on windows? and it's an extraordinarily large number njs: unclear; https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/dup-dup2 is murky on windows, handles are the real thing, and fds are this fake thing emulated in userspace by the CRT (= what they call libc). there's just a global array that maps from fd numbers to handles. So dup might just create a new entry in the array pointing to the same handle... also, https://github.com/twisted/twisted/pull/996#issuecomment-381213313 libuv doesn't bother to do any EMFILE recovery on windows (and for extra fun, the answer might be different for different versions of python, b/c they use different versions of the CRT) indeed :( it looks like on 3.6 at least os.dup does create new handles: https://paste.pound-python.org/show/Rm5xcGEzpZ0WXTd7hWNd/ runciter: the accept() docs do imply that WSAEMFILE is the way it reports out-of-handle errors, but ... I am certainly prepared to believe that they are lying but apparently _file_ handles, not socket handles njs: i am now pretty sure socket handles and file handles are very different things as evidenced by their different limits and the fact that there's EMFILE and WSAEMFILE (also in general all the WSA error codes are different because of some weird history I don't understand where winsockets was like this add on API rather than something built into the OS, and it was originally a spec rather than an implementation, so they had to define the ABI separately from the regular windows ABI, or... soemthing? it is all extremely weird) njs: winsock 2 evidently lets you turn a socket handle into a file handle with some performance penalty (see that link) err, the first msdn link it is super weird runciter: I believe in modern windows, normally, socket handles are real OS handles "modern windows" and "normally" are doing a lot of work in that sentence hah njs: i'm not sure there's one kind of "handle" in windows truly, i'm not sure the MSDN docs for winsock 1 and 2 imply they aren't but they're not super clear so there is a C type called HANDLE, which is an integer of a certain size and this is punned to cover both OS-level handles and "pseudo-handles" hmmm I think the hack is that there's a special bit that marks a HANDLE number as being a "pseudo-handle", which then gets special handling in user-space njs: do you have any helpful hints into MSDN docs on this? so like the console handles are traditionally pseudo-handles or is this accumulated street smarts? and winsock has its own thing, where traditionally SOCKETs are either a pointer to some user space emulation thing, or a real OS handle, and they've generally moved towards the latter over time (but both are still supported) yeah, this is accumulated from reading many many little hints buried in different places unfortunately nuts, i see njs: so one possibility is that in my tests i was somehow using user space emulated handles my tests did confirm that os.dup(0) is not sufficient to trigger WSAEMFILE on a subsequent accept, though I bet os.dup is hitting the fd limit before it hits the handle limit hmm, I tried doing 'while True: socks.append(socket.socket())' to see what happened and what happened is my terminal froze? windows is so weird yes! right? there's https://msdn.microsoft.com/en-us/library/windows/desktop/ms737647(v=vs.85).aspx which i was going to use to try and figure out what was happening haha wut "The sign-in process couldn't display security and sign-in options when Ctrl+Alt+Delete was pressed. If Windows doesn't respond, press Esc, or use the power switch to restart." yup mine spontaenously shut down so apparently the way windows reports out-of-handles errors is to blow up the world but then froze partway through I like the crash-only philosophy but maybe this is taking it too far to be fair it's pretty good at getting attention maybe this isn't anything we need to worry about then the default handle limits are definitely much higher than the fd limits you'll usually see on unix njs: that seems to be libuv's take on it "In one of the rare cases where Windows sets a hard-coded upper limit on a resource, the Executive defines 16,777,216 (16*1024*1024) as the maximum number of handles a process can allocate" -- https://blogs.technet.microsoft.com/markrussinovich/2009/09/29/pushing-the-limits-of-windows-handles/ ... so, like also: "The Microsoft Winsock provider limits the maximum number of sockets supported only by available memory on the local computer" -- https://msdn.microsoft.com/en-us/library/windows/desktop/ms739169(v=vs.85).aspx can you lower that? njs: there's a (drum roll) registry key supposedly I'm not seeing any evidence of that, no but i wasn't able to find it on windows 10 seems... bad? that might be unix brain damage there's lots of references to a 10k limit, but that seems to be a limit on the number of window objects, not a general limit on handles ```

It would still be good to: