Enable AF_UNIX support in Windows

[54b8045a-ddbb-4ba8-a97f-d3f1db53e980]

BPO	33408
Nosy	@pfmoore, @ericvsmith, @giampaolo, @tjguk, @zware, @zooba, @graingert, @animalize, @filips123, @paulmon, @websurfer5
PRs	python/cpython#14823

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-feature', '3.9', 'OS-windows'] title = 'Enable AF_UNIX support in Windows' updated_at = user = 'https://github.com/filips123' ``` bugs.python.org fields: ```python activity = actor = 'graingert' assignee = 'none' closed = False closed_date = None closer = None components = ['Windows'] creation = creator = 'filips123' dependencies = [] files = [] hgrepos = [] issue_num = 33408 keywords = ['patch'] message_count = 9.0 messages = ['316067', '347629', '347645', '347646', '348080', '348880', '348913', '348923', '348936'] nosy_count = 11.0 nosy_names = ['paul.moore', 'eric.smith', 'giampaolo.rodola', 'tim.golden', 'zach.ware', 'steve.dower', 'graingert', 'malin', 'filips123', 'Paul Monson', 'Jeffrey.Kintscher'] pr_nums = ['14823'] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue33408' versions = ['Python 3.9'] ```

[54b8045a-ddbb-4ba8-a97f-d3f1db53e980]

Unix socket (AF_UNIX) is now avalible in Windows 10 (April 2018 Update). Please add Python support for it. More details about it on https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/

[eef812a9-0c55-49e1-85e3-717a8588c91c]

I've been asked by my team to investigate what is required to enable AF_UNIX in Python.

Is anyone else actively investigating this? I did a prelinary investigation and the impact on test in test_sockets was pretty simple.

However there were 26 test failures in test_asyncio tests, with only the naive changes to all AF_UNIX to work with the socket class on Windows.

The first failure I looked at was caused by ProactorEventLoop in windows_events.py not providing a windows-specific version of create_unix_server. This results in the code falling back to AbstractServer.create_unix_server which raises NotImplementedError.

[b8e5a65c-bed4-4c5d-ba32-799f883ba638]

Have you upgraded the building SDK that supports AF_UNIX?

And should remove AF_UNIX flag at runtime on systems older than Windows 10 1804, see bpo-32394.

[b8e5a65c-bed4-4c5d-ba32-799f883ba638]

It would be nice to investigate the habit of using AF_UNIX in Python code on GitHub: https://github.com/search?l=Python&q=AF_UNIX&type=Code

If adding this flag will break a lot of code, due to lacking supports to datagram, maybe we should not add AF_UNIX at once, and waiting for full support to AF_UNIX.

[eef812a9-0c55-49e1-85e3-717a8588c91c]

I don't know if datagram support is coming to AF_UNIX on Windows.

The changes will only add the flag on Windows, and will enable stream sockets to use AF_UNIX on Windows. In mind this isn't breaking datagram support. It is true that it's a subset of what appears to be supported on Linux/Unix.

[b8e5a65c-bed4-4c5d-ba32-799f883ba638]

The current AF_UNIX address family of Windows10 doesn't support datagram, adding this flag may break some cross-platform code:

https://github.com/osbuild/osbuild/blob/9371eb9eaa3d0a7cab876eb4c7b70f519dfbd915/osbuild/__init__.py#L253 https://github.com/watsona4/dot_julia2/blob/5b7632b8089be01af706d8152555e711e0a81f06/conda/3/pkgs/python-3.7.3-h0371630_0/lib/python3.7/logging/handlers.py#L676 https://github.com/bognikol/Eleusis/blob/828672492b9cdb3444ddf466c0ac3055572277f7/Dependencies/02_macOS/40_gtk%2B/x64/lib/python2.7/test/test_socketserver.py#L177 https://github.com/sarnautov/python3/blob/005d75d8ac2b11a06707ce0184021a727d6fe844/lib/python3.6/test/test_asyncio/test_unix_events.py#L311 https://github.com/yanlianglai/ssr_python_leeblog/blob/8113fe3f32a250cc52e0cddad9761692dd840967/shadowsocks/manager.py#L60 ...

So I'm -1 on adding AF_UNIX personally.

I thought a compromise proposal:

• if (building SDK >= 1804 and run-time Windows >= 1804), add a undocumented flag _WIN_AF_UNIX, its value is equal to AF_UNIX.
• only use _WIN_AF_UNIX in stdlib, this is a black-box optimization.
• if someone really need high performance, he/she can uses this undocumented flag privately.

If one day Windows support full AF_UNIX, we can remove _WIN_AF_UNIX and add standard AF_UNIX.

[zooba]

Most of those examples would break today if run on Windows, though (AttributeError). So they'd just continue to break, probably with a different error (I'm not clear what happens if we specify SOCK_DGRAM with this change).

Having an undocumented field doesn't really help much - what we want is a documented field with a different name, perhaps WIN_AFUNIX? (I think we need to keep the "AF" prefix for real values and not mess up the namespace ourselves.) That way we can document that it matches AF_UNIX when defined on Windows, but will not trigger existing code that checks for the presence of AF_UNIX. Then we'll have to update the standard library to use either/both values where supported.

[eef812a9-0c55-49e1-85e3-717a8588c91c]

If you try to create a datagram socket with the current AF_UNIX changes on Windows the error is:

OSError: [WinError 10047] An address incompatible with the requested protocol was used

All of the examples given will fail to load with AttributeError on Windows because socket.AF_UNIX is not a valid attribute on Windows.

I think it's possible that there is code that detects whether AF_UNIX is an attribute on socket or not and does something different on Windows, but I haven't seen an example of it so far.

I can make the changes to WIN_AF_UNIX when I continue working on this, just in case there is portable code out there somewhere.

[zooba]

I think it's possible that there is code that detects whether AF_UNIX is an attribute on socket or not and does something different on Windows, but I haven't seen an example of it so far.

One of the test suites linked above had a skipUnless(hasattr(socket, 'AF_UNIX')) on the whole class, and at least one test assumed that it could use datagram support, so there's the example.

[futscdav]

This was really surprising to me. I have a native app that exposes a unix domain socket on windows for IPC, and wanted to write a Python client for it, only to find that the default socket interface does not allow using the AF_UNIX family on windows. Is there a 3rd party library that can work around this?

[robertlagrant]

@paulmon did you have any luck with this?

[paulmon]

This work wasn't completed because datagram support for AF_UNIX sockets on windows didn't exist when I made the original pull request.

I did a quick search and I don't think that has changed.

See this blog for more info about AF_UNIX support (not sure if the info is current or not) : https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/

[futscdav]

From what I understand, datagram support for AF_UNIX on windows is not planned. SOCK_STREAM is the only supported mode, but that's obviously already very useful in practice.

[leiless]

Any update on this?

[xavierog]

This issue stems from that blog post: https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/ That blog post's author stated that, contrary to what he had initially announced, abstract Unix sockets are not and will not be supported in Windows: https://github.com/microsoft/WSL/issues/4240#issuecomment-620805115

Overall, 2260 days after that blog post, as far as I know:

• AF_UNIX+SOCK_STREAM is less capable that announced/expected;
• there is no AF_UNIX+SOCK_DGRAM implementation in sight;
• SCM_RIGHTS was mentioned but not implemented;
• SOCK_SEQPACKET was not even mentioned;
• same for SCM_CREDENTIALS (but that mechanism is usually very OS-specific and Python does not exactly support it).

At this stage, and unless Microsoft publishes a new shiny blog post expressing their firm intent to further develop AF_UNIX sockets in their operating system, it seems fair to "time out" and proceed with an ad-hoc implementation.

I do not have any strong opinion regarding AF_UNIX vs WIN_AF_UNIX. I would suggest to add AF_UNIX_PARTIALSUPPORT and document that using this family may raise a specific exception (e.g. a class akin to NotImplementedError?). This way, AF_UNIX keeps its current meaning of "this system offers decent support for multiple flavours of AF_UNIX sockets" and AF_UNIX_PARTIALSUPPORT users know what to expect.

[ErikHons]

AF_UNIX+SOCK_STREAM would let dbus-next talk to the freedesktop reference server over a UDS like other clients. I believe all of the identification options would work fine there.

[lestie198]

@futscdav

This was really surprising to me. I have a native app that exposes a unix domain socket on windows for IPC, and wanted to write a Python client for it, only to find that the default socket interface does not allow using the AF_UNIX family on windows. Is there a 3rd party library that can work around this?

I have a similar issue, did you find a solution?

[xavierog]

Is there a 3rd party library that can work around this?

I have a similar issue, did you find a solution?

Theoretically, it should be possible to leverage ctypes[1] to reach the WinSock API and create AF_UNIX sockets. But I doubt the resulting socket could ever be shoehorned back into the socket module, meaning all other operations (send, receive, close, ...) would need to be done through ctypes+WinSock too. This is presumably a steep price compared to most practical applications.

[1] -- correct me if I am wrong but it seems the PyWin32 project does not expose the WinSock API.