Limit connection buffering

[matrss]

I've been experimenting with fowl for an idea I have in mind, and I noticed an issue: It seems like a listening port will happily accept tons of data, to the point that it can lock up the local machine, while only being able to slowly send that data out.

To reproduce: On one machine:

• iperf3 -s localhost -p 54321
• fowl <code> On another machine:
• fowl -L 12345:54321 <code>
• iperf3 -c localhost -p 12345 -n 1G

Then notice that the Gigabyte of data is written almost instantly, while sending will take quite a while. On my first try I left out the -n 1G part and managed to completely freeze my laptop with it.

I assume there is some kind of buffer on the connection that is a bit too large.

[meejah]

Yes, this would be a great feature.

Internally, I simply haven't yet hooked up any of the Twisted backpressure stuff (so in your "crash" case it was probably eating up all available memory).

That's "Producer / Consumer" if anyone reading this is thinking about a PR (that is, the relevant Twisted documentation to look at is IProducer / IConsumer interfaces and friends). This can be tuned, so ideally a command-line option could control how much buffering (note that this is "Python process-space buffering" and not anything the OS etc does already).

[matrss]

So, I have no clue about twisted, but what I am getting from the docs is that this would just be a case of adding a ProducerConsumerProxy, or maybe a ShapedConsumer, in the right place(s) and exposing the bufferSize via CLI arguments, right?

[meejah]

To give a better answer I'd need to remind myself of all the pieces there, but "something like that". It might actually be as easy as a .registerProducer() (or consumer) in the right spot since everything should already be "producer aware".