Creating a simple QUIC server and client to transfer data without HTTP

[a3y3]

Hi, I'm interested to use aioquic to compare the performance of TCP vs QUIC. Note that I don't really wish to use HTTP - I just want a QUIC python server that accepts a bunch of data from a client.

I have gone through the docs but unfortunately it's still not very clear to me how I should use the library to achieve what I want.

1. Can I use aioquic's QUIC API to achieve what I want?
2. If I am not using HTTP, do I still need to worry about certs and/or private keys?
3. Is there an example that creates a barebones QUIC server and client? If not, is there a flowchart I should refer to to get started?

I'm sorry if these questions seem to be too basic - in my head using QUIC on python would have been as simple as using a library to write a TCP like program in python - except the library handles the QUIC implementation. If aioquic isn't meant to serve that function, please close this issue. Otherwise, given a little help, I'd be glad to add to the examples or the docs.

[jlaine]

You can definitely leverage aioquic to build the kind of setup you describe. However, I suggest maybe starting with reading a little more information about QUIC before plunging into the code, as there seem to be several misunderstandings. These are mostly along the lines: TCP / QUIC are not functionally equivalent.

• QUIC provides (mandatory) encryption, so yes you do need to worry about certs / private keys.
• QUIC provides support for multiple independent streams of data within a single connection, so a "socket-like" API would not accurately cover it.
• A "QUIC server and client" doesn't really describe much: and you need some sort of protocol riding on top of QUIC, and this protocol needs to be announced / negotiated using ALPN.

I think a good starting point would be to look at the doq_client.py and doq_server.py examples provided with aioquic. These two examples implement a client + server for "DNS over QUIC", which is a very simple protocol. You can strip down these examples to plug in whatever you need. The key here is that your "QuicConnectionProtocol" subclass will receive events from the QUIC layer in its quic_event_received method.

[a3y3]

That's immensely helpful, thanks @jlaine! I still don't understand what you mean by having a protocol on top of QUIC - but that's probably my lack of knowledge right now about QUIC itself. I think this is a good starting point - thanks a lot!

Edit: Do feel free to close the issue if it's clogging up your issues tracker. I didn't close it myself in case I had more questions during the development (I will keep them to a minimum, I promise)

[a3y3]

I could get what I wanted by stripping down the DNS over QUIC examples. Thanks for giving me that lead! The research about QUIC itself was quite fascinating. Thanks a lot!

[jlaine]

Cool, glad I could help!

[a3y3]

Hi @jlaine - sorry for reopening the issue, but seems like I might still have questions about QUIC after all.

I'm trying to send an image from my client to my server. Since it's a single image, I figured a single stream should do the trick - and it does, except that when the server saves the file only half the image is visible.

I feel like I am doing something fundamentally wrong with the library. My understanding so far was that with QUIC, you create streams, and you can send data on each stream with send_stream_data(stream_id, data, end). However, there's also a send_datagram_frame(data) method. I tried sending data through this function, but on the Server side, I never seem to get a DatagramFrameReceived event.

Here's a snippet from my Server:

class ImgServerProtocol(QuicConnectionProtocol):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.f = None

    def quic_event_received(self, event: QuicEvent):
        if isinstance(event, DatagramFrameReceived):
            # this never gets triggered
            if self.f is None:
                print("Opening for the first time")
                self.f = open("big_file.png", 'w+b')

            data = event.data
            print("Writing data")
            self.f.write(data)

        if isinstance(event, StreamDataReceived):
            if self.f is None:
                print("Opening for the first time")
                self.f = open("big_file.png", 'w+b')

            data = event.data   
            print("Writing data with stream id", event.stream_id)
            self.f.write(data)

class SessionTicketStore:
    """
    Simple in-memory store for session tickets.
    """

    def __init__(self) -> None:
        self.tickets: Dict[bytes, SessionTicket] = {}

    def add(self, ticket: SessionTicket) -> None:
        self.tickets[ticket.ticket] = ticket

    def pop(self, label: bytes) -> Optional[SessionTicket]:
        return self.tickets.pop(label, None)

if __name__ == "__main__":
    # arg parser fluff, omitted

    configuration = QuicConfiguration(
        alpn_protocols=["img_transfer"],
        is_client=False,
        max_datagram_frame_size=65536,
        quic_logger=None,
    )

    configuration.load_cert_chain(args.certificate, args.private_key)

    ticket_store = SessionTicketStore()

    if uvloop is not None:
        uvloop.install()
    loop = asyncio.get_event_loop()
    loop.run_until_complete(
        serve(
            args.host,
            args.port,
            configuration=configuration,
            create_protocol=ImgServerProtocol,
            session_ticket_fetcher=ticket_store.pop,
            session_ticket_handler=ticket_store.add,
            retry=args.retry,
        )
    )
    try:
        loop.run_forever()
    except KeyboardInterrupt:
        pass

And here's the client:

class ImgClient(QuicConnectionProtocol):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    async def send_data(self, file_name) -> None:
        f = open(file_name, "rb")
        buffer_size = 2048
        data = f.read(buffer_size)
        stream_id = self._quic.get_next_available_stream_id() # create a single new stream
        while (data):
            # self._quic.send_datagram_frame(data)  # what does this method do?
            self._quic.send_stream_data(stream_id, data, False)
            self.transmit()
            print("Sending...")
            data = f.read(buffer_size)

async def run(
    configuration: QuicConfiguration,
    host: str,
    port: int
) -> None:
    print(f"Connecting to {host}:{port}")
    async with connect(
        host,
        port,
        configuration=configuration,
        create_protocol=ImgClient,
    ) as client:
        client = cast(ImgClient, client)
        await client.send_data("../../assets/big_file.png")

if __name__ == "__main__":
    # arg parser fluff, omitted

    configuration = QuicConfiguration(
        alpn_protocols=["img_transfer"], is_client=True, max_datagram_frame_size=65536
    )
    if args.ca_certs:
        configuration.load_verify_locations(args.ca_certs)
    if args.insecure:
        configuration.verify_mode = ssl.CERT_NONE
    if args.session_ticket:
        try:
            with open(args.session_ticket, "rb") as fp:
                configuration.session_ticket = pickle.load(fp)
        except FileNotFoundError:
            pass

    loop = asyncio.get_event_loop()
    loop.run_until_complete(
        run(
            configuration=configuration,
            host=args.host,
            port=args.port
        )
    )

To sum up: the problem I'm seeing is that when this code is run, the server only seems to save half the image. If what my understanding of data transmission over QUIC is completely incorrect, is there a resource you recommend I read up on?

[jlaine]

First off: sorry I had forgotten DoQ uses an unusual codepath: forget about send_datagram_frame and DatagramFrameReceived, this is clearly not what you want. You definitely want a stream, since your data needs to be reliably retransmitted and ordered.

Secondly: your client is shutting down too early. It's closing the connection before the data has actually been acknowledged by the server. You probably want a different logic, something along the lines of:

• the client sends out it's data, and sets the end_stream argument of send_stream_data to True on the last chunk
• the server closed the connection once it receives the final block (event.stream_ended will be True)

[a3y3]

Aah, that certainly makes sense! I removed the waiter functionality since I thought there isn't anything the client should be waiting for - I'll re-add it and check again. Thanks!

Also - doq doesn't really use DatagramFrame - I just found out about that when I was looking at other examples (siduck)

[a3y3]

@jlaine Thanks for your help! That worked, and the file does get transferred in full - although I'm seeing quite slow transfer speeds. Here's my updated client:

class ImgClient(QuicConnectionProtocol):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._ack_waiter: Optional[asyncio.Future[None]] = None

    async def send_data(self, file_name) -> None:
        f = open(file_name, "rb")
        buffer_size = 1048576
        data = f.read(buffer_size)
        stream_id = self._quic.get_next_available_stream_id()
        stream_end = False
        num_packets = math.ceil(os.path.getsize(file_name)/buffer_size)
        print("Size is", os.path.getsize(file_name),"buffer is", buffer_size, "num_packets=", num_packets)
        counter = 1
        while (data):
            if counter == num_packets:
                print("Set stream to True!")
                stream_end = True
            self._quic.send_stream_data(stream_id, data, stream_end)
            self.transmit()
            print("Sending...", counter)
            data = f.read(buffer_size)
            counter += 1

        waiter = self._loop.create_future()
        self._ack_waiter = waiter

        return await asyncio.shield(waiter)

    def quic_event_received(self, event: QuicEvent) -> None:
        if isinstance(event, StreamDataReceived):
            response = event.data
            print(response.decode())
            waiter = self._ack_waiter
            self._ack_waiter = None
            waiter.set_result(None)

And the server:

class ImgServerProtocol(QuicConnectionProtocol):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.f = None
        self.counter = 0

    def quic_event_received(self, event: QuicEvent):
        if isinstance(event, StreamDataReceived):
            if self.f is None:
                print("Opening for the first time")
                self.f = open("big_file.png", 'w+b')

            self.counter += 1
            data = event.data
            self.f.write(data)
            print("Wrote data: #", self.counter)
            if event.end_stream:
                self._quic.send_stream_data(event.stream_id, "Finished writing to file".encode(), True)
                self.f.close()
                print("Finished writing to file")

Whenever the server's quic_event_received is called I immediately increase the counter and print it, so I can see how many times it was called.

For an image of size 41026764 bytes (40MB), the server prints out Wrote data: # 32006 - this indicates that it received data in chunks of 41026764/32006 bytes, which equals around 1281 bytes.

I wrote a TCP+TLS program with a server buffer size of 1281 bytes and that data transfer happens in a 2 seconds - whereas the QUIC program takes around 8 seconds. Any ideas what could be going wrong?

[a3y3]

Seems like TCP is faster than QUIC...but only when the packet loss is less than 10%.

After 10% TCP's time to transfer increases exponentially, but QUIC's increases linearly, so QUIC beats TCP after 10% packet loss. Not sure why that is (@jlaine if you have any thoughts on why this would be, I'd absolutely love to hear them), but I think I have the results I need, so I'll close this issue!

[guest271314]

@a3y3 Did you manage to send data on the same stream? I am trying to send a stream of data from server to WebTransport client in browser yet have not been able to achieve the expected result.