twisted web client getPage is no longer available in V22

[jbleyel]

https://github.com/DrMichael/FritzCall/blob/master/src/FritzConnection.py#L39

You can change it like this:

OLD:

from twisted.web.client import getPage

getPage(url, method=b'GET').addCallback(self.successfunc).addErrback(self.errorfunc)

NEW:

from twisted.internet.reactor import callInThread
from requests import get, exceptions

    callInThread(self.threadGetPage, url, self.successfunc, self.errorfunc)

    def threadGetPage(self, link, success, fail):
        try:
            response = get(link)
            response.raise_for_status()
        except exceptions.RequestException as error:
            fail(error)
        else:
            success(response.content)

[DrMichael]

Thanks for the hint. Isn't there a more elegant way to replace it? :-)

And what about post, agent, headers?

[jbleyel]

something like that.

from twisted.internet.reactor import callInThread
from requests import post, exceptions

    callInThread(self.threadPostPage, url, self.successfunc, self.errorfunc)

    def threadPostPage(self, link, success, fail):
        try:
            response = post(link, headers={...})
            response.raise_for_status()
        except exceptions.RequestException as error:
            fail(error)
        else:
            success(response.content)

https://requests.readthedocs.io/en/latest/

Please note!. For python 2 you need an older version of requests and this older version may not have all features. But I think this is not a problem in your plugin.

[DrMichael]

Hi, how would I call the successfunc with an additional parameter? Trying to translate: getPage(six.ensure_binary(url)).addCallback(self._getCalls_cb2, callback)

[dhwz]

I disagree you don't need requests you can replace that with twisted agent which is fully compatible with the old code

[DrMichael]

Do you have an example?

[dhwz]

Sure That's the code used in MediaPortal. You can reuse it in FritzCall just give proper credits. You may need to cleanup it a bit as you may only need twAgentGetPage

[jbleyel]

def getPage(url):
    def sendUrl(url, timeout, headers):
        r = requests.get(url, headers=headers, verify=False, timeout=timeout)
        r.raise_for_status()
        return r.content
    headers = {}
    timeout = 5
    return threads.deferToThread(lambda: sendUrl(url, timeout, headers))

[DrMichael]

https://app.box.com/s/9qi9mm9pek787cu6c73rgus9ec5sqbcy That's the code used in MediaPortal. You can reuse it in FritzCall just give proper credits.

Not available any more...

[dhwz]

@DrMichael https://app.box.com/s/cbesu2e3qme402zf5v4y6p212gsgvyu2

[DrMichael]

HI,

I guess, I need help... (Perhaps I am blind). I replaced getPage with one using deferToThread. That is working fine, except in FritzConnection.py I cannot get it working.

I replaced getPage in FritzConnection.py and changed the results to get the content from the response. But I get an 500 internal server error. I think to remember, that I had it in the past already. MIght be an issue with the FritzBox...

Perhaps you have an idea.

Michael https://www.dropbox.com/s/agzkxqsxxmvn2bd/enigma2-plugin-extensions-fritzcall_20230107-rev1632_all.ipk?dl=0 https://www.dropbox.com/s/5dyteqqjuzodhih/FritzCall_20230107_rev1632.tar.gz?dl=0

[jbleyel]

You probably mean this: https://github.com/DrMichael/FritzCall/blob/master/src/FritzCallFBF.py#L2560

This is a POST request

Mockup ->

def _getPage(url, params=None, headers=None, cookies=None):
    return deferToThread(requests.post, url, params=params, headers=headers, timeout=30.05)

parms = urlencode({'getpage': '../html/login_sid.xml'})
headers={'Content-Type': "application/x-www-form-urlencoded", 'Content-Length': str(len(parms))},
url = "http://%s/cgi-bin/webcm" % (config.plugins.FritzCall.hostname.value)
self.debug("'" + url + "' parms: '" + parms + "'")

_getPage(url, parms, headers).addCallback(lambda x: self._md5Login(callback, x)).addErrback(lambda x: self._oldLogin(callback, x))

[DrMichael]

Ah, that works, my getPage in __init.py can do both. Everything works except the calls in FritzConnection.py. I get a 500 when executing FritzAction.execute...

I have made a new branch https://github.com/DrMichael/FritzCall/tree/New_getPage to have a look at.

Could it be, because there is a soapaction in the headers?

[DrMichael]

Ok, good hint. The POST did not come through due to coding differences. Now it gets through. But I get back an UPNP error:

<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<s:Fault>
<faultcode>s:Client</faultcode>
<faultstring>UPnPError</faultstring>
<detail>
<UPnPError xmlns="urn:schemas-upnp-org:control-1-0">
<errorCode>401</errorCode>
<errorDescription>Invalid Action</errorDescription>
</UPnPError>
</detail>
</s:Fault>
</s:Body>
</s:Envelope>

The action has not changed and looks good: 'urn:dslforum-org:service:DeviceConfig:1#X_AVM-DE_CreateUrlSID'. Probably again something with text vs. binary vs. unicode?

[DrMichael]

Ah, should mention: all testing done so far under OpenATV 7.1 with python 3.10.4

[jbleyel]

Sounds very good. Nice Job!!!

[DrMichael]

Ah, that in the new branch is not yet working... The communication FritzConnection.py is still throwing errors! It would be great, if someone could figure out, why...

[jbleyel]

Can you post the error details and a shot howto reproduce the error?

[DrMichael]

Well, not possible to reproduce without FritzBox... I attached the following:

• the diff between the old working version with the old getPage and the current not working version with the new getPage using deferTpThread
• the logs old and new
• the response from the FB signalling "invalid action".

I could also provide the headers and body of the call, which results in invalid action... FritzConnection.zip

[jbleyel]

I think I cannot help here because I have no Fritzbox

[DrMichael]

So, got it. Phew. Wrong kw parameter for requests.post. Master updated, the other branch will be removed.