OSError: [WinError 6] The handle is invalid

[swfiua]

This code:

    import multiprocessing
    from curio.channel import Channel
    a, b = multiprocessing.Pipe()
    aa = Channel.from_Connection(a)

Throws this exception:

Traceback (most recent call last): File "piper.py", line 7, in aa = Channel.from_Connection(a) File "c:\users\john.gill\devel\curio\curio\channel.py", line 59, in from_Connection reader = FileStream(open(conn._handle, 'rb', buffering=0)) OSError: [WinError 6] The handle is invalid

It means curio.run_in_process() doesn't work on windows.

Afraid I have no idea what is going on here, opening the _handle from a multiprocessing.Pipe() on windows seems to be a no-no.

[swfiua]

Just an addenda to this.

I discovered there is some magic in the msvcrt that converts the connection handle into something open seems to be able to digest:

handle = msvcrt.open_osfhandle(conn._handle, os.O_RDWR)

But this doesn't get us much further as os.set_blocking() doesn't exist on windows.

Seems "OVERLAPPED" is what you need on Windows.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365683(v=vs.85).aspx

Starting to think there is no way this is going to work on Windows :(

[njsmith]

[it sounds like you've probably figured this out, but for the record:]

Yeah, curio is being a bit naughty and making assumptions about how multiprocessing.Pipe is implemented internally, but it's implemented differently on Windows and elsewhere. See multiprocessing/connection.py -- there's a big if sys.platform != "win32": ... else: ....

On non-Windows, Pipe is backed by either an os.pipe or an os.socketpair. Either way curio is happy.

On Windows, Pipe is backed by a "named pipe", which is a Windows thing. You definitely can't use selectors or do non-blocking I/O on named pipes; you have to switch to IOCP or one of Windows' other more exotic I/O APIs. There's some discussion about possibly implementing these for curio in #68, but with the way the kernel is currently written, yeah, there's no quick hack that's going to make multiprocessing.Pipe work.

I can also explain the open_osfhandle thing: Windows doesn't natively have a concept of "file descriptors" (!). Instead it has "handles". Handles are integers referring to some kernel object, just like file descriptors on Unix, but they're different in some ways (e.g. file descriptors are required to count up from 0, file descriptors refer to files while handles are more general), so the Windows equivalent of libc, which is known as the "CRT", emulates file descriptors on top of handles -- it keeps a little array in userspace that says "fd 0 is handle 12, fd 1 is handle 834, ...". So CRT functions like open and close and read and write return or accept a file descriptors, then internally convert them into handles to actually do the work, and native windows functions like CreateFile and CloseHandle and ReadFile and WriteFile take handles directly. msvcrt.open_osfhandle takes a handle and assigns it an fd; msvcrt.get_osfhandle does the inverse operation.

This is important to know about for a few reasons:

• The fd table in the CRT is limited to a maximum of 2048 entries. There isn't any corresponding limit on the number of handles you can have open.
• Obviously if you have a random small integer representing a file you need to know whether it's a fd or a handle. Python doesn't really make this easy. os functions work on fds, and if you do open(...) and then call fileno() it'll give you an fd. But if you do socket(...) and call fileno() it'll give you a handle. If trying to understand the guts of libraries like subprocess or multiprocessing you just have to trace through all the calls to figure out which kind of integer you have...

[swfiua]

Thanks for the explanations - much appreciated.

[dabeaz]

run_in_process() might not work on Windows, but you should be able to use run_in_executor() instead. Create a ProcessPoolExecutor executor from concurrent.futures and then use the Curio run_in_executor() function on it to execute a CPU-bound task.

That said, we should still look into making run_in_process() work. Even if it means just using something really basic. Right now, it is specifically written for Unix systems.

[dabeaz]

I'd add a note here... the reason Curio doesn't use normal thread and process executors (from concurrent.futures) is that their behavior is too ill-defined to work in the presence of timeouts and task cancellation. The Curio versions try to handle this in a more sane and predictable way. I suspect we could build a Window's compatible run_in_process() function by adapting the run_in_thread() function to use native multiprocessing features. The reason it doesn't do this now is that multiprocessing uses backing threads to coordinate I/O. The Curio version is free-thread and uses Curio's built-in I/O. Of course, sadly that means it only works with I/O that's compatible with the selectors module. Window pipes don't work.