Running multiple get requests returns connection lost on all but the first request

[Column01]

Hey there,

I'm relatively new to Twisted's way of programming and still earning deferrals, but I've recently started to maintain some code using twisted.

I've added treq as a dependency and I'm in the process of updating all getPage requests to a proper web request using treq.get but I'm now having issues where my second and all other attempts return the following error to my error handler (which just prints it out at the moment)

The code that is making the requests is found here. This is called from another module where the data is from a treq.get().json() deferral from here. I'm wondering, is it because I'm making the calls from inside a deferral?

The run looks like the following in the console with the current print output from the code, to me, it looks like the second request for 1.17.1-snapshot fails (which is consistent with the lack of its presence in the final output before the error.

Output:

server@server-desktop:~/testing$ mark2 jar-list
Getting info for: 1.17.1-release with URL: https://launchermeta.mojang.com/v1/packages/8b976413591b4132fc4f27370dcd87ce1e50fb2f/1.17.1.json
Getting info for: 1.17.1-snapshot with URL: https://launchermeta.mojang.com/v1/packages/16eaa3c82ff61349c340cad27c40478114a7136b/1.17.1-rc2.json
The following server jars/zips are available:
  vanilla-release | Vanilla Release
Unhandled error in Deferred:

Traceback (most recent call last):
Failure: twisted.web._newclient.ResponseFailed: [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>, <twisted.python.failure.Failure twisted.web.http._DataLoss: >]

[Column01]

Hmm, this is most certainly due to me making multiple requests asynchronously.

Changing to use inline callbacks has fixed this issue.

Any tips on how I can do this async without breaking treq? I assume I need to make a client... which is what I was avoiding by using treq.

[Column01]

hmm. Having issues downloading a file now... Response is: <treq.response._Response 200 'application/octet-stream' 43,626,592 bytes> but it runs d_result.callback((filename, content)) with an empty content presumably due to an error in grabbing the content?

def jar_get(name):
    d_result = defer.Deferred()

    @defer.inlineCallbacks
    def got_results(results):
        for r in results:
            if name == '-'.join(r.name_short):
                resp = yield treq.get(r.url)
                print(resp)
                print(r.url)
                filename = r.url.split('/')[-1]
                content = yield resp.content()
                d_result.callback((filename, content))

        d_result.errback(Exception("{} is not available!".format(name)))

    d = get_raw()
    d.addCallbacks(got_results, d_result.errback)
    return d_result

EDIT: Same error as last time, connection just gets closed.

[Column01]

Hmm managed to get it working. My deferrals were running their callbacks early. Think I sorted it out for now