New addition of vSockets to the python socket module

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

BPO	27584
Nosy	@gpshead, @ncoghlan, @tiran, @bitdancer, @berkerpeksag, @kushaldas, @caavery
PRs	python/cpython#2489
Files	vsocket.patch: vsockets for linux support vsock_rev2.patch: vsockets for linux support revision 2 nc-vsock 0001-bpo-27584-New-addition-of-vSockets-to-the-python-soc.patch REAME.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['3.7', 'type-feature', 'library'] title = 'New addition of vSockets to the python socket module' updated_at = user = 'https://github.com/caavery' ``` bugs.python.org fields: ```python activity = actor = 'christian.heimes' assignee = 'none' closed = True closed_date = closer = 'christian.heimes' components = ['Library (Lib)'] creation = creator = 'Cathy Avery' dependencies = [] files = ['43816', '45411', '45964', '46977', '46978'] hgrepos = ['349'] issue_num = 27584 keywords = ['patch'] message_count = 32.0 messages = ['270935', '272358', '272363', '272366', '272403', '280423', '282798', '283534', '283535', '283607', '283608', '283612', '283620', '296410', '296413', '297064', '297065', '297276', '297285', '297516', '297568', '297835', '298165', '298166', '298167', '298169', '298171', '298173', '298174', '298805', '298953', '301527'] nosy_count = 7.0 nosy_names = ['gregory.p.smith', 'ncoghlan', 'christian.heimes', 'r.david.murray', 'berker.peksag', 'kushal.das', 'Cathy Avery'] pr_nums = ['2489'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue27584' versions = ['Python 3.7'] ```

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

I have added AF_VSOCK support to python's 3.6 socket module ( socketmodule.c socketmodule.h cloned from https://hg.python.org/cpython ). The implementation is very similar to AF_NETLINK. AF_VSOCK requires the VMware-specific VMCI transport which is currently upstream or the virtio-vsock drivers developed by Stefan Hajnoczi at Red Hat. The virtio-vsock drivers are not upstream yet but more information with source and build instructions can be found at http://qemu-project.org/Features/VirtioVsock.

More information on vSocket programming can be found at https://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/ws9_esx60_vmci_sockets.pdf

The VMCI transport supports SOCK_DGRAM and SOCK_STREAM on both Linux and Windows. Virtio-vsock currently supports SOCK_STREAM only on Linux.

My python addition supports SOCK_STREAM and SOCK_DGRAM calls on Linux only.

I have tested my implementation on both driver sets on Linux.

Attached is a diff file so you can see which files I've modified. These include a new configure.ac. I have already tested the new file generation by running autoreconf.

Also included in the patch is an updated socket.rst file however I could not get the final html page to be double spaced.

[bitdancer]

Looks like there's a missing versionadded directive in the doc patch.

Is it possible/sensible to add tests for the new feature?

(I haven't reviewed the patch in detail, hopefully someone with more experience with C socket programming than I have will do that.)

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

Sure I can add tests. I would like to base them on the existing socket tests. Where are those?

I did add a version

+ .. versionadded:: 3.4

It just not may not be the right one.

[bitdancer]

Ah, I see. No, the versionadded will be 3.6, and should go *after* the documentation of the new socket type.

The existing socket tests are in Lib/test/test_socket.py.

[kushaldas]

The patch can be applied, and build successfully. I have ran the current test suite[1]. The two failed tests do not seem to be have anything to do with this patch (read the end of the consoleText output). I think the thing remaining is the new test cases, and NEWS file update.

[1] https://ci.centos.org/job/cPython-build-patch/24/consoleText

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

Please forgive the long delay in providing this update. I got a little sidetracked. Attached is the patch for Python 3.7. It includes fixes suggested in rev 1 plus VSOCK tests in test_socket.py.

Thanks,

Cathy

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

Is there anything else that is needed for this patch?

Thanks!

[bitdancer]

The second patch seems to be missing the configure changes. Also, the tests have some over-long lines (we limit line lengths to 79 characters). I realize there are other long lines in that file, but no need to add more :)

There is trailing whitespace on a number of lines in your patch.

Since this is new, we may not want to accept it until the support hits upstream. Specifically, it will be difficult to get a review if the reviewer has to build a custom kernel to test the code :) You do say that the VMCI is upstream, but I don't know what that means. Which upstream?

Note: I'm not familiar with the socket C code, so I haven't reviewed the C code changes. The tests look fine to me.

For the docs, the proposal doesn't seem to follow the format of the existing docs. I would expect only the first paragraph located where you have it. The remaining constants should be in the 'module contents'/'constants' section, I think. Yes, that means each one gets a '.. versionadded' label. Presumably also an 'availablility' label with whatever the minimum kernel version is...another reason we may need to wait.

[bitdancer]

Oh, I see, the ac changes are there, I was looking at the patch delta instead of the complete patch.

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

Is there a format checker I could use on the patch?

VMCI and the vmw_vsock_vmci_transport kernel modules are located in the upstream linux tree at

git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

They have been there for about years. These drivers are part of various downstream kernels such as RHEL. You will need a Vmware virtual machine in order to test it.

Only the virtio-vsock driver is a new vsock application that needs to be custom built.

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

Sorry about the typo the drivers have been there for about 4 years.

[bitdancer]

'make patchcheck' will do whitespace checking because that's hard to eyeball (although many editors/IDEs do support making it visible nowadays). We don't use any other checking tools other than eyeballs, since not all of the existing code conforms to PEP7/8 and for various reasons we aren't going to update most of the old code to conform.

So, if I'm running an ubuntu virtual machine under VMWare Fusion (which I already have set up) I should be able to get the tests to run? Or does it need to be RedHat (or presumably CentOS)?

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

First make sure the driver is in your kernel. It will be with RHEL. Look in /lib/modeles/"your kernel name"/kernel/net/vmw_vsock/vmw_vsock_vmci_transport. I have never tried it on vmware fusion. I have tested it on ESX. See if there is a VMCI option to enable on your VM's settings. Start the vm and do an lsmod to see if vmw_vsock_vmci_transport is loaded.

I've attached a little C program thats netcat for vsock. Its a quick confirmation that your transport is loaded correctly. It will show you your CID.

run ./nc-vsock

CID = 973033371
CID = 0x39ff4f9b
usage: ./nc-vsock [-l <port> [-t <dst> <dstport>] | <cid> <port>]

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

The vsock code is now in the linux upstream kernel and qemu. I will be resubmitting my vsock patches for python. So from what I can tell the tip of the devel tree is for 3.7 and that the source control has switched to git. My question is do I need a github account or can I push git patches directly to this issues page?

Thanks,

Cathy

[berkerpeksag]

You don't need a GitHub account for contributing to CPython, but the pull request workflow is the preferred way. You can still attach your patches to this issue.

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

I've attached the third version of VSOCK patch addressing the concerns of the last rev. I've also included a README file that lists instructions on how to setup a test environment.

Thanks

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

Help file.

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

I also issued a pull request

https://github.com/python/cpython/pull/2489

Let me know if I screwed it up.

Thanks,

Cathy

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

The build and test bots failed but I don't understand why. The build could not find module _socket but that is not new and other modules failed.

The test could not import fcntl which I did add but should not fail as I have run this test many time and other tests import fcntl.

https://github.com/python/cpython/pull/2489

[ncoghlan]

I'm attempting to figure out whether or not we have a buildbot in the Buildbot fleet that will cover this test case.

Based on the pre-merge CI run, it seems Ubuntu 14.04 is too old to include the required kernel headers.

However, it looks like RHEL/CentOS are also currently still missing the userspace changes to fully enable AF_VSOCK support (as the Red Hat backport flow appears to have gone through the dedicated hypervisor variant first): https://bugzilla.redhat.com/show_bug.cgi?id=1315822

So it's looking to me like we're going to need either a recent Fedora, a non-LTS Ubuntu, or a Debian 9 system to be confident we have the right headers available.

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

Fedora 25 has the proper headers.

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

So I revised my code based on the reviews and I passed all the checks ... now what?

Thanks,

Cathy

[bitdancer]

I think we are waiting on confirmation that we have a buildbot that has the necessary headers.

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

OK thanks!

[gpshead]

I updated my PGO buildbot to Debian 9 which should have them.

http://buildbot.python.org/all/builders/AMD64%20Debian%20PGO%203.x

\~$ grep AF_VSOCK /usr/include///* /usr/include/x86_64-linux-gnu/bits/socket.h:#define AF_VSOCK PF_VSOCK

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

You will also need linux/vm_sockets.h in order to build.

[gpshead]

yep, linux/vm_sockets.h exists. I believe it's kernel headers are from 4.9.

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

That should do it.

[gpshead]

Everything compiles successfully on this host - configure detects the header has HAVE_LINUX_VM_SOCKETS_H set to 1.

test_socket passes on this host. However since i'm not running with a /dev/vsock, the unittests (correctly) skip the new tests.

testCreateSocket (test.test_socket.BasicVSOCKTest) ... skipped 'VSOCK sockets required for this test.' testCrucialConstants (test.test_socket.BasicVSOCKTest) ... skipped 'VSOCK sockets required for this test.' testSocketBufferSize (test.test_socket.BasicVSOCKTest) ... skipped 'VSOCK sockets required for this test.' testVSOCKConstants (test.test_socket.BasicVSOCKTest) ... skipped 'VSOCK sockets required for this test.' testStream (test.test_socket.ThreadedVSOCKSocketStreamTest) ... skipped 'VSOCK sockets required for this test.'

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

Hi,

I believe I am waiting for a final review. Is there anything else I need to be doing at this point.

Thanks,

Cathy

[ef2a15ff-9f10-4c7a-b808-0084f5c92983]

There is an outstanding review on my pull request at https://github.com/python/cpython/pull/2489 as there is an red X at changes requested by kushaldas and I believe I have made the necessary changes.

Again please let me know if there is anything that I need to do as I am new to this process.

Thanks,

Cathy

[tiran]

New changeset effc12f8e9e20d0951d2ba5883587666bd8218e3 by Christian Heimes (caavery) in branch 'master': bpo-27584: New addition of vSockets to the python socket module (bpo-2489) https://github.com/python/cpython/commit/effc12f8e9e20d0951d2ba5883587666bd8218e3