Protocols support 🚀

[joamag]

Description

Since the inception of the asyncio library in Python 3.5, the Python network world has changed, and a lot of "scalability" emerged from that.

As Netius was created before the arrival of asyncio, the tech approach to network diverged. Although both strategies represent nonblocking asyncio event-loop-based solutions overall, they have different approaches and abstractions for server and client implementation.

The objective of this issue is to merge (via compatibility layer) both approaches, making it possible for the following scenarios:

• A netius HTTP should be able to run on the asyncio event loop
• An asyncio service should be able to run in the netius event loop

Meaning full bi-directional compatibility between Netius's way of working and the asyncio one.

Implementation

Architecture

• Transport - Connection (Stream) vs Connectionless (Datagram)
• Protocol - Equivalent to part of the current Connection and also part of the current Server/Client (protocol logic)
• Server - Equivalent to the newly created Service that spawns multiple connections (that should be converted into transports)
• Parsers - Already exists and should remain pretty much the same
• Event Loop - Must be improved to distance itself from the server vs client approach, meaning that an event loop is something simpler that only handles the polling strategy adapting details
• Servers/Clients (Agents) - Must be simple Abstract API entry-points that facilitate some operations (no logic), this abstract logic should be re-use as much as possible

Change ideas

Command-based client

We must rethink what a command-based client (e.g., SMTP, IMAP, etc.) is, as their session is currently very hard-coded. It must be much more similar to a command-based server, kind of like a bot-based server (more similar to the FTP server). And their state machine should be more flexible.

Migration strategy

This is the overall idea of how to migrate a client.

HTTPConnection -> HTTPProtocol

Principles

• Retro-compatibility should always be a priority
• Concepts like auto_close and auto_pause no longer make sense
• Bi-directional compatibility between netius and asyncio
  • Protocols developed for asyncio should be able to run without problems in netius
  • Netius server implementations should be able to run without problem in the base asyncio event loop

Testing

To test that both strategies are working, one can use the ASYNCIO=1 env variable to run in the asyncio event loop. The COMPAT=1 mode will ensure that Netius's services and clients run the compatibility layer, meaning that they should be able to run with the asyncio API behavior expectations, calling the proper protocol callbacks, etc.

Reference

Transports and protocols (callback-based API) aiohttp server

[joamag]

It's important to make support for a simple server like this one echos_tcp.py as soon as possible.