zerotier / libzt

Encrypted P2P sockets over ZeroTier
https://zerotier.com
Other
173 stars 53 forks source link

Align Python bindings with existing interfaces and Python best practices #169

Open bostonrwalker opened 2 years ago

bostonrwalker commented 2 years ago

Overview

Context

The existing Python bindings are a thin wrapper over a Swig interface generated on top of the C++ library. This approach has the following limitations:

These limitations exclude libzt from achieving "mature" Python package status and becoming more widely adopted.

Goal

With a harmonized API, better testing, and better documentation, libzt sockets can be used as a drop-in replacement anywhere that Python sockets are currently used. This will simplify the process of extending existing Python libraries and protocols to work over ZeroTier and will lay the groundwork for a flourishing ecosystem.

Changes

Socket interface

Align libzt Python socket interface with Python's own IP sockets as provided in cpython/socket.py. All function signatures, behaviours, and exception flows should be identical whenever possible. When not possible, libzt should be designed to be as semantically similar as possible in order to maximize compatibility. Access to full functionality of libzt should always be preserved.

Node management

Management of the ZeroTier node state should be simplified using the context manager pattern or similar. Singleton and global variables should be used to reflect the fact that only one libzt node can be managed per process.

Static functionality

Availability of all static libzt functionality should be verified and provided where it does not already exist.

Exceptions

Exception flow should match whenever possible, with libzt taking advantage of built-in exceptions. This is largely possible since both libraries are based on LwIP.

Testing

A comprehensive Python test suite will be developed to guarantee this functionality. As many existing socket.py test cases as possible should be ported over and applied to libzt.

Type hinting

Since libzt and the Python community in general will no longer support Python 2, all methods will be updated with fully-developed Python 3 type hinting.

Documentation

Docstrings will be updated and augmented to use the Sphinx standard. Sphinx will be used to automatically generate comprehensive documentation.

The current README.md should be updated to explain all features, functionality, and design goals of the Python interface.

Examples

The examples should be updated to reflect the new functionality and simplified interface. A complicated-looking example is a deterrent to potential users who are looking to get their feet wet.

Plan of work

Socket interface

socket Methods

Method cpython (3.10.2) libzt (1.8.4) Target signature libzt signature Signature aligned Type hinting Docstring Implementation aligned Exceptions aligned Issue
__init__ socket.py#L220 sockets.py#L55 __init__(self, family: Union[AddressFamily, int, None] = AddressFamily.AF_INET, type: Union[SocketKind, int, None] = SocketKind.SOCK_STREAM, proto: Union[IpProto, int, None] = IPProto.IPPROTO_UNSPEC, fileno: Optional[int] = None) __init__(self, sock_family=-1, sock_type=-1, sock_proto=-1, sock_fd=None) ✔️ Unknown TODO
__enter__ socket.py#L236 __enter__(self) -> socket TODO
__exit__ socket.py#L239 __exit__(self, exc_type: Optional[Type[BaseException]], exc_val: Optional[BaseException], exc_tb: Optional[TracebackType]) -> bool TODO
__repr__ socket.py#L243 __repr__(self) -> str TODO
__getstate__ socket.py#L272 __getstate__(self) -> NoReturn TODO
dup socket.py#L275 socket.py#L272 dup(self) -> socket dup(self) ✔️ TODO
accept socket.py#L286 socket.py#L219 accept(self) -> Tuple[socket, Union[str, Tuple[str, int]]] accept(self) ✔️ ✔️ Unknown TODO
makefile socket.py#L302 socket.py#L328 makefile(self, mode: str = "r", buffering: Optional[int] = None, *, encoding: Optional[str] = None, errors: Optional[str] = None, newline: Optional[str] = None) -> io.IOBase makefile(mode="r", buffering=None, *, encoding=None, errors=None, newline=None) ✔️ Unknown #146
sendfile socket.py#L465 socket.py#L409 sendfile(self, file: io.IOBase, offset: int = 0, count: Optional[int] = None) sendfile(self, file, offset=0, count=None) ✔️ Unknown TODO
close socket.py#L498 socket.py#L239 close(self) -> None close(self) ✔️ ✔️ Unknown TODO
detach socket.py#L504 socket.py#L266 detach(self) -> int detach(self) ✔️ Unknown TODO
family socket.py#L515 socket.py#L70 @property family(self) -> AddressFamily @property family(self) ✔️ Unknown TODO
type socket.py#L521 socket.py#L75 @property type(self) -> SocketKind @property type(self) ✔️ Unknown TODO
proto Exposed from socketmodule.c#L988 socket.py#L80 @property proto(self) -> IPProto @property proto(self) ✔️ Unknown TODO
get_inheritable socket.py#L532 socket.py#L281 get_inheritable(self) -> bool get_inheritable(self) ✔️ Unknown Unknown TODO
set_inheritable socket.py#L534 socket.py#L413 set_inheritable(self, inheritable: bool) -> bool set_inheritable(self, inheritable) ✔️ Unknown Unknown TODO

Native socket methods

Method cpython (3.10.2) libzt (1.8.4) Target signature(s) libzt signature Signature aligned Type hinting Docstring Implementation aligned Exceptions aligned Issue
setblocking socketmodule.c#L2775 socket.py#L417 setblocking(self, flag: bool, /) -> None setblocking(self, flag) ✔️ Unknown Unknown TODO
getblocking socketmodule.c#L2802 socket.py#L285 getblocking(self) -> bool getblocking(self) ✔️ Unknown Unknown TODO
settimeout socketmodule.c#L2866 (dev) socket.py#L423 settimeout(self, value: Optional[float], /) -> None settimeout(self, value) ✔️ ✔️ ✔️ Unknown #143
gettimeout socketmodule.c#L2914 (dev) socket.py#L440 gettimeout(self) -> Optional[float] gettimeout(self) ✔️ ✔️ ✔️ Unknown #143
setsockopt socketmodule.c#L2941 socket.py#L427 setsockopt(self, level: int, optname: int, value: Union[int, ReadableBuffer], /) setsockopt(self, level: int, optname: int, value: None, optlen: int, /) setsockopt(self, level, optname, value) ✔️ Unknown TODO
getsockopt socketmodule.c#L3029 socket.py#L299 getsockopt(self, level: int, optname: int, buflen: Optional[int] = None, /) -> Union[int, bytes, bytearray] getsockopt(self, level, optname, buflen=None) ✔️ ✔️ Unknown TODO
bind socketmodule.c#L3098 socket.py#L231 bind(self, address: Address, /) -> None bind(self, local_address) Unknown Unknown TODO
connect socketmodule.c#L3268 socket.py#L247 connect(self, address: Address, /) -> None connect(self, address) ✔️ ✔️ Unknown TODO
connect_ex socketmodule.c#L3299 socket.py#L255 connect_ex(self, address: Address, /) -> int connect_ex(self, address) ✔️ ✔️ Unknown TODO
fileno socketmodule.c#L3330 socket.py#L276 fileno(self) -> int fileno(self) ✔️ ✔️ ✔️ TODO
getsockname socketmodule.c#L3344 socket.py#L295 getsockname(self) -> Optional[Address] getsockname(self) ✔️ TODO
getpeername socketmodule.c#L3374 socket.py#L291 getpeername(self) -> Optional[Address] getpeername(self) ✔️ TODO
listen socketmodule.c#L3404 socket.py#L311 listen(self, backlog: Optional[int] = None, /) -> None listen(self, backlog=None) ✔️ ✔️ Unknown TODO
recv socketmodule.c#L3493 socket.py#L332 recv(self, bufsize: int, flags: Union[MsgFlag, int, None] = None, /) -> bytes recv(self, n_bytes, flags=0) ✔️ Unknown TODO
recv_into socketmodule.c#L3542 socket.py#L365 recv_into(self, buffer: WritableBuffer, nbytes: Union[int, None] = 0, flags: Union[MsgFlag, int, None] = None, /) -> int recv_into(self, buffer, n_bytes, flags) TODO
recvfrom socketmodule.c#L3677 socket.py#L349 recvfrom(self, bufsize: int, flags: Union[MsgFlag, int, None] = None, /) -> Tuple[bytes, Address] recvfrom(self, bufsize, flags) TODO
recvfrom_into socketmodule.c#L3729 socket.py#L361 recvfrom_into(self, buffer: WritableBuffer, nbytes: Optional[int] = 0, flags: Union[MsgFlag, int, None] = None, /) -> Tuple[int, Address] recvfrom_into(self, buffer, n_bytes, flags) TODO
recvmsg socketmodule.c#L3957 socket.py#L353 recvmsg(self, bufsize: int, ancbufsize: Optional[int] = 0, flags: Union[MsgFlag, int, None] = None, /) -> Tuple[bytes, List[Tuple[int, int, bytes]], int, Optional[Address]] recvmsg(self, bufsize, ancbufsize, flags) TODO
recvmsg_into socketmodule.c#L4024 socket.py#L357 recvmsg_into(self, buffers: Iterable[WritableBuffer], ancbufsize: Optional[int] = 0, flags: Union[MsgFlag, int, None] = None, /) -> Tuple[int, List[Tuple[int, int, bytes]], int, Optional[Address]] recvmsg_into(self, buffers, ancbufsize, flags) TODO
send socketmodule.c#L4135 socket.py#L369 send(self, bytes: ReadableBuffer, flags: Union[MsgFlag, int, None] = None, /) -> int send(self, data, flags=0) ✔️ Unknown TODO
sendall socketmodule.c#L4171 (dev) socket.py#L385 sendall(self, bytes: ReadableBuffer, flags: Union[MsgFlag, int, None] = None, /) -> None sendall(self, bytes, flags=0) ✔️ ✔️ ✔️ Unknown #144
sendto socketmodule.c#L4276 socket.py#L389 sendto(self, bytes: ReadableBuffer, flags: Union[MsgFlag, int], address: Address, /) -> int sendto(self, bytes: ReadableBuffer, address: Address, /) -> int sendto(self, n_bytes, flags, address) TODO
sendmsg socketmodule.c#L4419 socket.py#L393 sendmsg(self, buffers: Iterable[ReadableBuffer], ancdata: Optional[Iterable[Tuple[int, int, ReadableBuffer]]] = None, flags: Union[MsgFlag, int, None] = None, address: Optional[Address] = None, /) -> int sendto(self, n_bytes, flags, address) TODO
sendmsg_afalg socketmodule.c#L4623 socket.py#L397 sendmsg_afalg(self, msg: Optional[Iterable[ReadableBuffer]] = None, /, *, op: int, iv: Optional[ReadableBuffer] = None, assoclen: Optional[int] = None, flags: Union[MsgFlag, int, None] = None) -> int sendmsg_afalg(self, msg, *, op, iv, assoclen, flags) TODO
ioctl socketmodule.c#L4814 socket.py#L307 ioctl(self, control: int, option: Union[int, Tuple[int, int, int]], /) -> int ioctl(self, request, arg=0, mutate_flag=True) Unknown Unknown TODO
share socketmodule.c#L4870 share(self, process_id: int, /) -> bytes Not known if possible to implement
shutdown socketmodule.c#L4790 socket.py#L434 shutdown(self, how: int, /) -> None shutdown(self, how) ✔️ ✔️ Unknown TODO

Socket module functions

Function cpython (3.10.2) libzt (1.8.4) Target signature libzt signature Signature aligned Type hinting Docstring Implementation aligned Exceptions aligned Issue
fromfd socket.py#L539 socket.py#L105 fromfd(fd: Optional[int], family: Union[AddressFamily, int, None] = AddressFamily.AF_INET, type: Union[SocketKind, int, None] = SocketKind.SOCK_STREAM, proto: Union[IpProto, int, None] = IPProto.IPPROTO_UNSPEC) fromfd(self, fd, sock_family, sock_type, sock_proto=0) TODO
send_fds socket.py#L539 socket.py#L401 send_fds(sock: socket, buffers: Iterable[ReadableBuffer], fds: Iterable[int], flags: Union[MsgFlag, int, None] = None, address: Optional[Address] = None) send_fds(self, sock, buffers, fds, flags, address) TODO
recv_fds socket.py#L563 socket.py#L405 recv_fds(sock: socket, bufsize: int, maxfds: int, flags: Union[MsgFlag, int, None] = None) recv_fds(self, sock, bufsize, maxfds, flags) TODO
fromshare socket.py#L583 socket.py#L109 fromshare(info: bytes) -> socket fromshare(self, data) Not known if possible to implement
socketpair socket.py#L594 socket.py#L84 socketpair(family: Union[AddressFamily, int, None] = AddressFamily.AF_UNIX, type: Union[SocketKind, int, None] = SocketKind.SOCK_STREAM, proto: Union[IPProto, int, None] = IPProto.IPPROTO_UNSPEC) -> Tuple[socket, socket] socketpair(self, sock_family, sock_type, sock_proto) Not known if possible to implement
getfqdn socket.py#L779 socket.py#L119 getfqdn(name: Optional[str] = None) -> str getfqdn(self, name) TODO
create_connection socket.py#L808 socket.py#L90 create_connection(address: INETAddress, timeout: Union[float, Object, None] = socket._GLOBAL_DEFAULT_TIMEOUT, source_address: Optional[INETAddress] = None) -> socket # Note: socket._GLOBAL_DEFAULT_TIMEOUT is used as a sentinal here with a specific meaning of: "use the default timeout" create_connection(self, remote_address) TODO
has_dualstack_ipv6 socket.py#L853 socket.py#L65 def has_dualstack_ipv6(): bool has_dualstack_ipv6(self) TODO
create_server socket.py#L869 socket.py#L97 create_server(address: INETAddress, *, family: Union[AddressFamily, int] = AddressFamily.AF_INET, backlog: Optional[int] = None, reuse_port: bool = False, dualstack_ipv6: bool = False) -> socket create_server(self, local_address, sock_family=libzt.ZTS_AF_INET, backlog=None) TODO
getaddrinfo socket.py#L938 socket.py#L183 getaddrinfo(host: Optional[str], port: Union[int, str, None], family: Union[AddressFamily, int, None] = AddressFamily.AF_UNSPEC, type: Union[SocketKind, int, None] = None, proto: Union[IPProto, int, None] = IPProto.IPPROTO_UNSPEC, flags: Optional[int] = None) -> List[Tuple[AddressFamily, SocketKind, int, Optional[str], Address]] getaddrinfo(self, host, port, sock_family=0, sock_type=0, sock_proto=0, flags=0) TODO

Native socket module functions

Function cpython (3.10.2) libzt (1.8.4) Target signature(s) libzt signature Signature aligned Type hinting Docstring Implementation aligned Exceptions aligned Issue
gethostname socketmodule.c#L5371 socket.py#L131 gethostname() -> str gethostname(self) TODO
sethostname socketmodule.c#L5437 socket.py#L203 sethostname(name: str, /) -> None sethostname(self, name) TODO
gethostbyname socketmodule.c#L5477 socket.py#L123 gethostbyname(hostname: str, /) -> str gethostbyname(self, hostname) TODO
gethostbyname_ex socketmodule.c#L5649 socket.py#L127 gethostbyname_ex(hostname: str, /) -> Tuple[str, List[str], List[str]] gethostbyname_ex(self, hostname) TODO
gethostbyaddr socketmodule.c#L5723 socket.py#L135 gethostbyaddr(ip_address: str, /) -> Tuple[str, List[str], List[str]] gethostbyaddr(self, ip_address) TODO
getservbyname socketmodule.c#L5821 socket.py#L147 getservbyname(servicename: str, protocolname: Optional[str] = None, /) -> int getservbyname(self, servicename, protocolname) TODO
getservbyport socketmodule.c#L5856 socket.py#L147 getservbyport(port: int, protocolname: Optional[str] = None, /) -> str getservbyport(self, port, protocolname) TODO
getprotobyname socketmodule.c#L5897 socket.py#L143 getprotobyname(protocolname: str, /) -> int getprotobyname(self, protocolname) TODO
ntohs socketmodule.c#L6088 socket.py#L159 ntohs(x: int, /) -> int ntohs(self, x) TODO
ntohl socketmodule.c#L6117 socket.py#L155 ntohl(x: int, /) -> int ntohl(self, x) TODO
htons socketmodule.c#L6151 socket.py#L167 htons(x: int, /) -> int htons(self, x) TODO
htonl socketmodule.c#L6180 socket.py#L163 htonl(x: int, /) -> int htonl(self, x) TODO
inet_aton socketmodule.c#L6221 socket.py#L171 inet_aton(ip_string: str, /) -> bytes inet_aton(ip_string) TODO
inet_ntoa socketmodule.c#L6292 socket.py#L175 inet_ntoa(packed_ip: ReadableBuffer, /) -> str inet_ntoa(packed_ip) TODO
inet_pton socketmodule.c#L6318 socket.py#L179 inet_pton(address_family: Union[AddressFamily, int], ip_string: str, /) -> bytes inet_pton(address_family, ip_string) TODO
inet_ntop socketmodule.c#L6374 socket.py#L183 inet_ntop(address_family: Union[AddressFamily, int], packed_ip: ReadableBuffer, /) -> str inet_ntop(address_family, packed_ip) TODO
getnameinfo socketmodule.c#L6557 socket.py#L139 getnameinfo(sockaddr: Address, flags: Optional[int] = None, /) -> Tuple[str, int] getnameinfo(self, sockaddr, flags) TODO
getdefaulttimeout socketmodule.c#L6658 socket.py#L195 getdefaulttimeout() -> Optional[float] getdefaulttimeout(self) TODO
setdefaulttimeout socketmodule.c#L6677 socket.py#L199 setdefaulttimeout(timeout: Optional[float], /) -> None setdefaulttimeout(self, timeout) TODO
if_nameindex socketmodule.c#L6700 socket.py#L207 if_nameindex() -> List[Tuple[int, str]] if_nameindex(self) TODO
if_nametoindex socketmodule.c#L6786 socket.py#L211 if_nametoindex(if_name: str, /) -> int if_nametoindex(self, if_name) TODO
if_indextoname socketmodule.c#L6815 socket.py#L215 if_indextoname(if_index: int, /) -> str if_indextoname(self, if_index) TODO
CMSG_LEN socketmodule.c#L6848 socket.py#L187 CMSG_LEN(length: int, /) -> int CMSG_LEN(length) TODO
CMSG_SPACE socketmodule.c#L6878 socket.py#L191 CMSG_SPACE(length: int, /) -> int CMSG_SPACE(length) TODO

SocketIO object

The socket.SocketIO class defined at socket.py#L662 will be extended for use with the libzt socket.makefile() function (see #146).

Exceptions

Enums and typing

Node state management

A Session or similar class will be developed that will greatly simplify starting a node and connecting to a network:

with libzt.Session(net_id=0x0123456789abcdef):
    conn = libzt.socket()
    ...

ZeroTierNode will be changed to a singleton class to reflect the current one node per process limitation of libzt.

Both of these changes should retain backwards compatability.

Prioritization

Group 1 - Already partially/fully implemented

Method / function Type Existing impl? urllib3 requests socketserver Werkzeug aiohttp websocket-client gunicorn pyOpenSSL httplib2
socket.__init__ Methods ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
socket.accept Methods ✔️ ✔️ ✔️
socket.close Methods ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
socket.family Methods ✔️
socket.get_inheritable Methods ✔️
socket.proto Methods ✔️
socket.type Methods ✔️
socket.bind Methods (native) ✔️ ✔️ ✔️ ✔️ ✔️
socket.connect Methods (native) ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
socket.connect_ex Methods (native) ✔️ ✔️
socket.fileno Methods (native) ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
socket.getblocking Methods (native) ✔️
socket.getsockopt Methods (native) ✔️
socket.gettimeout Methods (native) ✔️ ✔️ ✔️ ✔️
socket.ioctl Methods (native) ✔️
socket.listen Methods (native) ✔️ ✔️ ✔️ ✔️
socket.recv Methods (native) ✔️ ✔️ ✔️ ✔️
socket.send Methods (native) ✔️ ✔️ ✔️ ✔️
socket.sendall Methods (native) ✔️ ✔️ ✔️ ✔️
socket.setblocking Methods (native) ✔️ ✔️
socket.setsockopt Methods (native) ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
socket.settimeout Methods (native) ✔️ ✔️ ✔️
socket.shutdown Methods (native) ✔️ ✔️ ✔️ ✔️ ✔️
create_connection Module functions ✔️
create_server Module functions ✔️
has_dualstack_ipv6 Module functions ✔️

Group 2 - High overlap with commonly used networking libraries

Method / function Type Existing impl? urllib3 requests socketserver Werkzeug aiohttp websocket-client gunicorn pyOpenSSL httplib2
socket.recvfrom Methods (native) ✔️
socket.sendto Methods (native) ✔️
getdefaulttimeout Module functions (native) ✔️
inet_aton Module functions (native) ✔️ ✔️ ✔️
inet_ntoa Module functions (native) ✔️ ✔️
socket.makefile Methods ✔️
socket.set_inheritable Methods ✔️ ✔️
socket.getpeername Methods (native) ✔️
socket.getsockname Methods (native) ✔️ ✔️ ✔️ ✔️
socket.recv_into Methods (native) ✔️
fromfd Module functions ✔️ ✔️
getaddrinfo Module functions ✔️ ✔️
getfqdn Module functions ✔️
gethostbyname Module functions (native) ✔️
getnameinfo Module functions (native) ✔️
inet_pton Module functions (native) ✔️

Group 3 - Low overlap, low implementation complexity

Method / function Type Existing impl? urllib3 requests socketserver Werkzeug aiohttp websocket-client gunicorn pyOpenSSL httplib2
socket.__enter__ Methods
socket.__exit__ Methods
socket.__getstate__ Methods
socket.__repr__ Methods
gethostbyname_ex Module functions (native)
inet_ntop Module functions (native)
setdefaulttimeout Module functions (native)
CMSG_LEN Module functions (native)
CMSG_SPACE Module functions (native)
gethostbyaddr Module functions (native)
gethostname Module functions (native)
getprotobyname Module functions (native)
getservbyname Module functions (native)
getservbyport Module functions (native)
htonl Module functions (native)
htons Module functions (native)
sethostname Module functions (native)
ntohl Module functions (native)
ntohs Module functions (native)

Group 4 - Low overlap, high implementation complexity

Method / function Type Existing impl? urllib3 requests socketserver Werkzeug aiohttp websocket-client gunicorn pyOpenSSL httplib2
socket.dup Methods
socket.sendfile Methods
socket.recvfrom_into Methods (native)
socket.recvmsg Methods (native)
socket.recvmsg_into Methods (native)
socket.sendmsg Methods (native)
recv_fds Module functions
send_fds Module functions

Group 5 - Probably not possible to implement

Method / function Type Existing impl? urllib3 requests socketserver Werkzeug aiohttp websocket-client gunicorn pyOpenSSL httplib2
socket.detach Methods
socket.sendmsg_afalg Methods (native)
socket.share Methods (native)
fromshare Module functions
socketpair Module functions
if_indextoname Module functions (native)
if_nameindex Module functions (native)
if_nametoindex Module functions (native)
joseph-henry commented 2 years ago

I definitely support this effort. Thank you.

I'll continue to test and merge your PRs as they come. It's nice to get a set of experienced eyes on the Python portion of this library. I'll try not to step on your toes but let me know if there's anything you need help working through.

Also, please reach out to us via our ticketing system so we can send something as a token of our gratitude. Just link to this issue.

bostonrwalker commented 2 years ago

Thanks! I couldn't find a way to send a private message on your ticketing system so I reached out on reddit.