RequestTraversalAgent not practically usable with IBodyProducers

[exarkun]

Data produced by an IBodyProducer will mostly just sit in a FakeTransport buffer (after https://twistedmatrix.com/trac/ticket/9003 is fixed, anyway). It might get delivered somewhere by a manual pump call but since pump is outside of the IAgent interface it's very difficult to convince anything to actually call it.

The consequence is a request made with RequestTraversalAgent and an IBodyProducer will probably hang indefinitely.

[exarkun]

Put another way, this test fails:

from treq.testing import RequestTraversalAgent
from twisted.web.resource import Resource
from twisted.trial.unittest import TestCase

class FooTests(TestCase):
    def test_foo(self):
        agent = RequestTraversalAgent(Resource())
        self.successResultOf(agent.put(b"http://foo/bar", data="baz"))

because the Deferred has no result yet - when there is no reason it couldn't have its result synchronously. The reason it doesn't have its result is that agent puts the data into a FileBodyProducer which uses cooperate to spread out the work of reading from the BytesIO that "baz" gets stuffed in to.

[glyph]

Part of the problem here (by no means unfixable, but I think why it wasn't originally fixed) is the use of "adapt to IBodyProducer" as the idiom for doing this translation. This means there's no immediately straightforward way to pass the reactor or reactor-alike that the tests want to use along to the various implementations here which want to do callLater or other reactor interfacing.

[exarkun]

Darn. I used RequestTraversalAgent in yet another test suite and once again the test can't work because of this behavior.

[exarkun]

Darn. I used RequestTraversalAgent in yet another test suite and once again the test can't work because of this behavior.

[exarkun]

It looks like there is a partial work-around for this. Instead of using RequestTraversalAgent, use StubTreq. They are not API compatible but StubTreq will at least handle request body strings synchronously.

[twm]

There is another workaround: use twisted.trial.unittest.TestCase instead of SynchronousTestCase (and let it spin the real reactor).

[exarkun]

Uh, I guess. Writing unit tests that do real I/O with the global reactor has a very 2009 feel to it, though, and one might question whether the solution is worse than the problem.

[twm]

FWIW, I usually work around this in test suites by either:

• Using a totally synchronous body producer like the BytesProducer in the narrative docs, when all request bodies are guaranteed to be small.
• Constructing the FileBodyProducer with a cooperator that has a terminationPredicateFactory that always returns False so that it synchronously exhausts its input.

Of course neither of these strategies work for files, since you don't control the FileBodyProducer.