Open bostonrwalker opened 2 years ago
I definitely support this effort. Thank you.
I'll continue to test and merge your PRs as they come. It's nice to get a set of experienced eyes on the Python portion of this library. I'll try not to step on your toes but let me know if there's anything you need help working through.
Also, please reach out to us via our ticketing system so we can send something as a token of our gratitude. Just link to this issue.
Thanks! I couldn't find a way to send a private message on your ticketing system so I reached out on reddit.
Overview
Context
The existing Python bindings are a thin wrapper over a Swig interface generated on top of the C++ library. This approach has the following limitations:
socket
class contains stubs for the full Python socket interface, but most of these methods are not implemented yetThese limitations exclude libzt from achieving "mature" Python package status and becoming more widely adopted.
Goal
With a harmonized API, better testing, and better documentation, libzt sockets can be used as a drop-in replacement anywhere that Python sockets are currently used. This will simplify the process of extending existing Python libraries and protocols to work over ZeroTier and will lay the groundwork for a flourishing ecosystem.
Changes
Socket interface
Align libzt Python socket interface with Python's own IP sockets as provided in cpython/socket.py. All function signatures, behaviours, and exception flows should be identical whenever possible. When not possible, libzt should be designed to be as semantically similar as possible in order to maximize compatibility. Access to full functionality of libzt should always be preserved.
Node management
Management of the ZeroTier node state should be simplified using the context manager pattern or similar. Singleton and global variables should be used to reflect the fact that only one libzt node can be managed per process.
Static functionality
Availability of all static libzt functionality should be verified and provided where it does not already exist.
Exceptions
Exception flow should match whenever possible, with libzt taking advantage of built-in exceptions. This is largely possible since both libraries are based on LwIP.
Testing
A comprehensive Python test suite will be developed to guarantee this functionality. As many existing socket.py test cases as possible should be ported over and applied to libzt.
Type hinting
Since libzt and the Python community in general will no longer support Python 2, all methods will be updated with fully-developed Python 3 type hinting.
Documentation
Docstrings will be updated and augmented to use the Sphinx standard. Sphinx will be used to automatically generate comprehensive documentation.
The current README.md should be updated to explain all features, functionality, and design goals of the Python interface.
Examples
The examples should be updated to reflect the new functionality and simplified interface. A complicated-looking example is a deterrent to potential users who are looking to get their feet wet.
Plan of work
Socket interface
socket
Methods__init__
__init__(self, family: Union[AddressFamily, int, None] = AddressFamily.AF_INET, type: Union[SocketKind, int, None] = SocketKind.SOCK_STREAM, proto: Union[IpProto, int, None] = IPProto.IPPROTO_UNSPEC, fileno: Optional[int] = None)
__init__(self, sock_family=-1, sock_type=-1, sock_proto=-1, sock_fd=None)
__enter__
__enter__(self) -> socket
__exit__
__exit__(self, exc_type: Optional[Type[BaseException]], exc_val: Optional[BaseException], exc_tb: Optional[TracebackType]) -> bool
__repr__
__repr__(self) -> str
__getstate__
__getstate__(self) -> NoReturn
dup
dup(self) -> socket
dup(self)
accept
accept(self) -> Tuple[socket, Union[str, Tuple[str, int]]]
accept(self)
makefile
makefile(self, mode: str = "r", buffering: Optional[int] = None, *, encoding: Optional[str] = None, errors: Optional[str] = None, newline: Optional[str] = None) -> io.IOBase
makefile(mode="r", buffering=None, *, encoding=None, errors=None, newline=None)
sendfile
sendfile(self, file: io.IOBase, offset: int = 0, count: Optional[int] = None)
sendfile(self, file, offset=0, count=None)
close
close(self) -> None
close(self)
detach
detach(self) -> int
detach(self)
family
@property family(self) -> AddressFamily
@property family(self)
type
@property type(self) -> SocketKind
@property type(self)
proto
@property proto(self) -> IPProto
@property proto(self)
get_inheritable
get_inheritable(self) -> bool
get_inheritable(self)
set_inheritable
set_inheritable(self, inheritable: bool) -> bool
set_inheritable(self, inheritable)
Native socket methods
setblocking
setblocking(self, flag: bool, /) -> None
setblocking(self, flag)
getblocking
getblocking(self) -> bool
getblocking(self)
settimeout
settimeout(self, value: Optional[float], /) -> None
settimeout(self, value)
gettimeout
gettimeout(self) -> Optional[float]
gettimeout(self)
setsockopt
setsockopt(self, level: int, optname: int, value: Union[int, ReadableBuffer], /)
setsockopt(self, level: int, optname: int, value: None, optlen: int, /)
setsockopt(self, level, optname, value)
getsockopt
getsockopt(self, level: int, optname: int, buflen: Optional[int] = None, /) -> Union[int, bytes, bytearray]
getsockopt(self, level, optname, buflen=None)
bind
bind(self, address: Address, /) -> None
bind(self, local_address)
connect
connect(self, address: Address, /) -> None
connect(self, address)
connect_ex
connect_ex(self, address: Address, /) -> int
connect_ex(self, address)
fileno
fileno(self) -> int
fileno(self)
getsockname
getsockname(self) -> Optional[Address]
getsockname(self)
getpeername
getpeername(self) -> Optional[Address]
getpeername(self)
listen
listen(self, backlog: Optional[int] = None, /) -> None
listen(self, backlog=None)
recv
recv(self, bufsize: int, flags: Union[MsgFlag, int, None] = None, /) -> bytes
recv(self, n_bytes, flags=0)
recv_into
recv_into(self, buffer: WritableBuffer, nbytes: Union[int, None] = 0, flags: Union[MsgFlag, int, None] = None, /) -> int
recv_into(self, buffer, n_bytes, flags)
recvfrom
recvfrom(self, bufsize: int, flags: Union[MsgFlag, int, None] = None, /) -> Tuple[bytes, Address]
recvfrom(self, bufsize, flags)
recvfrom_into
recvfrom_into(self, buffer: WritableBuffer, nbytes: Optional[int] = 0, flags: Union[MsgFlag, int, None] = None, /) -> Tuple[int, Address]
recvfrom_into(self, buffer, n_bytes, flags)
recvmsg
recvmsg(self, bufsize: int, ancbufsize: Optional[int] = 0, flags: Union[MsgFlag, int, None] = None, /) -> Tuple[bytes, List[Tuple[int, int, bytes]], int, Optional[Address]]
recvmsg(self, bufsize, ancbufsize, flags)
recvmsg_into
recvmsg_into(self, buffers: Iterable[WritableBuffer], ancbufsize: Optional[int] = 0, flags: Union[MsgFlag, int, None] = None, /) -> Tuple[int, List[Tuple[int, int, bytes]], int, Optional[Address]]
recvmsg_into(self, buffers, ancbufsize, flags)
send
send(self, bytes: ReadableBuffer, flags: Union[MsgFlag, int, None] = None, /) -> int
send(self, data, flags=0)
sendall
sendall(self, bytes: ReadableBuffer, flags: Union[MsgFlag, int, None] = None, /) -> None
sendall(self, bytes, flags=0)
sendto
sendto(self, bytes: ReadableBuffer, flags: Union[MsgFlag, int], address: Address, /) -> int
sendto(self, bytes: ReadableBuffer, address: Address, /) -> int
sendto(self, n_bytes, flags, address)
sendmsg
sendmsg(self, buffers: Iterable[ReadableBuffer], ancdata: Optional[Iterable[Tuple[int, int, ReadableBuffer]]] = None, flags: Union[MsgFlag, int, None] = None, address: Optional[Address] = None, /) -> int
sendto(self, n_bytes, flags, address)
sendmsg_afalg
sendmsg_afalg(self, msg: Optional[Iterable[ReadableBuffer]] = None, /, *, op: int, iv: Optional[ReadableBuffer] = None, assoclen: Optional[int] = None, flags: Union[MsgFlag, int, None] = None) -> int
sendmsg_afalg(self, msg, *, op, iv, assoclen, flags)
ioctl
ioctl(self, control: int, option: Union[int, Tuple[int, int, int]], /) -> int
ioctl(self, request, arg=0, mutate_flag=True)
share
share(self, process_id: int, /) -> bytes
shutdown
shutdown(self, how: int, /) -> None
shutdown(self, how)
Socket module functions
fromfd
fromfd(fd: Optional[int], family: Union[AddressFamily, int, None] = AddressFamily.AF_INET, type: Union[SocketKind, int, None] = SocketKind.SOCK_STREAM, proto: Union[IpProto, int, None] = IPProto.IPPROTO_UNSPEC)
fromfd(self, fd, sock_family, sock_type, sock_proto=0)
send_fds
send_fds(sock: socket, buffers: Iterable[ReadableBuffer], fds: Iterable[int], flags: Union[MsgFlag, int, None] = None, address: Optional[Address] = None)
send_fds(self, sock, buffers, fds, flags, address)
recv_fds
recv_fds(sock: socket, bufsize: int, maxfds: int, flags: Union[MsgFlag, int, None] = None)
recv_fds(self, sock, bufsize, maxfds, flags)
fromshare
fromshare(info: bytes) -> socket
fromshare(self, data)
socketpair
socketpair(family: Union[AddressFamily, int, None] = AddressFamily.AF_UNIX, type: Union[SocketKind, int, None] = SocketKind.SOCK_STREAM, proto: Union[IPProto, int, None] = IPProto.IPPROTO_UNSPEC) -> Tuple[socket, socket]
socketpair(self, sock_family, sock_type, sock_proto)
getfqdn
getfqdn(name: Optional[str] = None) -> str
getfqdn(self, name)
create_connection
create_connection(address: INETAddress, timeout: Union[float, Object, None] = socket._GLOBAL_DEFAULT_TIMEOUT, source_address: Optional[INETAddress] = None) -> socket # Note: socket._GLOBAL_DEFAULT_TIMEOUT is used as a sentinal here with a specific meaning of: "use the default timeout"
create_connection(self, remote_address)
has_dualstack_ipv6
def has_dualstack_ipv6(): bool
has_dualstack_ipv6(self)
create_server
create_server(address: INETAddress, *, family: Union[AddressFamily, int] = AddressFamily.AF_INET, backlog: Optional[int] = None, reuse_port: bool = False, dualstack_ipv6: bool = False) -> socket
create_server(self, local_address, sock_family=libzt.ZTS_AF_INET, backlog=None)
getaddrinfo
getaddrinfo(host: Optional[str], port: Union[int, str, None], family: Union[AddressFamily, int, None] = AddressFamily.AF_UNSPEC, type: Union[SocketKind, int, None] = None, proto: Union[IPProto, int, None] = IPProto.IPPROTO_UNSPEC, flags: Optional[int] = None) -> List[Tuple[AddressFamily, SocketKind, int, Optional[str], Address]]
getaddrinfo(self, host, port, sock_family=0, sock_type=0, sock_proto=0, flags=0)
Native socket module functions
gethostname
gethostname() -> str
gethostname(self)
sethostname
sethostname(name: str, /) -> None
sethostname(self, name)
gethostbyname
gethostbyname(hostname: str, /) -> str
gethostbyname(self, hostname)
gethostbyname_ex
gethostbyname_ex(hostname: str, /) -> Tuple[str, List[str], List[str]]
gethostbyname_ex(self, hostname)
gethostbyaddr
gethostbyaddr(ip_address: str, /) -> Tuple[str, List[str], List[str]]
gethostbyaddr(self, ip_address)
getservbyname
getservbyname(servicename: str, protocolname: Optional[str] = None, /) -> int
getservbyname(self, servicename, protocolname)
getservbyport
getservbyport(port: int, protocolname: Optional[str] = None, /) -> str
getservbyport(self, port, protocolname)
getprotobyname
getprotobyname(protocolname: str, /) -> int
getprotobyname(self, protocolname)
ntohs
ntohs(x: int, /) -> int
ntohs(self, x)
ntohl
ntohl(x: int, /) -> int
ntohl(self, x)
htons
htons(x: int, /) -> int
htons(self, x)
htonl
htonl(x: int, /) -> int
htonl(self, x)
inet_aton
inet_aton(ip_string: str, /) -> bytes
inet_aton(ip_string)
inet_ntoa
inet_ntoa(packed_ip: ReadableBuffer, /) -> str
inet_ntoa(packed_ip)
inet_pton
inet_pton(address_family: Union[AddressFamily, int], ip_string: str, /) -> bytes
inet_pton(address_family, ip_string)
inet_ntop
inet_ntop(address_family: Union[AddressFamily, int], packed_ip: ReadableBuffer, /) -> str
inet_ntop(address_family, packed_ip)
getnameinfo
getnameinfo(sockaddr: Address, flags: Optional[int] = None, /) -> Tuple[str, int]
getnameinfo(self, sockaddr, flags)
getdefaulttimeout
getdefaulttimeout() -> Optional[float]
getdefaulttimeout(self)
setdefaulttimeout
setdefaulttimeout(timeout: Optional[float], /) -> None
setdefaulttimeout(self, timeout)
if_nameindex
if_nameindex() -> List[Tuple[int, str]]
if_nameindex(self)
if_nametoindex
if_nametoindex(if_name: str, /) -> int
if_nametoindex(self, if_name)
if_indextoname
if_indextoname(if_index: int, /) -> str
if_indextoname(self, if_index)
CMSG_LEN
CMSG_LEN(length: int, /) -> int
CMSG_LEN(length)
CMSG_SPACE
CMSG_SPACE(length: int, /) -> int
CMSG_SPACE(length)
SocketIO object
The
socket.SocketIO
class defined at socket.py#L662 will be extended for use with the libztsocket.makefile()
function (see #146).Exceptions
error
class (alias for OSError)gaierror
class (exception raised for getaddrinfo/getnameinfo errors, a subclass of socket.error)herror
class (exception raised for gethostby* errors, a subclass of socket.error)timeout
class (alias for TimeoutError)Enums and typing
zts_errno
IntEnum to track socket error codes (Done - see #150)AddressFamily
IntEnumSocketKind
IntEnumNameInfo
IntEnumMsgFlag
IntEnumIPProto
IntEnum (see https://bugs.python.org/issue29515)INETAddress
,INET6Address
types (supported formats for passing asaddress
tobind()
depending on socket address family)Address
type, union of above typesReadableBuffer
type, union ofbytes
,bytearray
,array.array
, and 'memoryview[bytes/bytearray/array.array]' (see https://github.com/python/typing/issues/593)WritableBuffer
type, subclass ofReadableBuffer
type and union ofbytearray
,array.array
, and 'memoryview[bytearray/array.array]' (see https://github.com/python/typing/issues/593)Node state management
A
Session
or similar class will be developed that will greatly simplify starting a node and connecting to a network:ZeroTierNode
will be changed to a singleton class to reflect the current one node per process limitation of libzt.Both of these changes should retain backwards compatability.
Prioritization
Group 1 - Already partially/fully implemented
socket.__init__
socket.accept
socket.close
socket.family
socket.get_inheritable
socket.proto
socket.type
socket.bind
socket.connect
socket.connect_ex
socket.fileno
socket.getblocking
socket.getsockopt
socket.gettimeout
socket.ioctl
socket.listen
socket.recv
socket.send
socket.sendall
socket.setblocking
socket.setsockopt
socket.settimeout
socket.shutdown
create_connection
create_server
has_dualstack_ipv6
Group 2 - High overlap with commonly used networking libraries
socket.recvfrom
socket.sendto
getdefaulttimeout
inet_aton
inet_ntoa
socket.makefile
socket.set_inheritable
socket.getpeername
socket.getsockname
socket.recv_into
fromfd
getaddrinfo
getfqdn
gethostbyname
getnameinfo
inet_pton
Group 3 - Low overlap, low implementation complexity
socket.__enter__
socket.__exit__
socket.__getstate__
socket.__repr__
gethostbyname_ex
inet_ntop
setdefaulttimeout
CMSG_LEN
CMSG_SPACE
gethostbyaddr
gethostname
getprotobyname
getservbyname
getservbyport
htonl
htons
sethostname
ntohl
ntohs
Group 4 - Low overlap, high implementation complexity
socket.dup
socket.sendfile
socket.recvfrom_into
socket.recvmsg
socket.recvmsg_into
socket.sendmsg
recv_fds
send_fds
Group 5 - Probably not possible to implement
socket.detach
socket.sendmsg_afalg
socket.share
fromshare
socketpair
if_indextoname
if_nameindex
if_nametoindex