Switching from requests to aiohttp

[kyriog]

I'm offering this PR for switching from requests to aiohttp to add asyncio compatibility with this lib, as requested by #7 (and my personal needs).

Please note this is currently a work in progress, the scripts part haven't been updated yet as well as the tests, which will require to change the betamax dependency (because betamax is compatible only with requests).

I just wanted to share what I've already done, and wanted to know if you're interested by this kind of change.

How to use?

Here's some code example:

async def main():
    try:
        auth_mgr = await AuthenticationManager.from_file('tokens.json')
    except FileNotFoundError:
        auth_mgr = await AuthenticationManager.create()
        auth_mgr.email_address = "pyxb-testing@outlook.com"
        auth_mgr.password = "password"
    await auth_mgr.authenticate()
    await auth_mgr.close()  # required to properly close aiohttp.ClientSession

    client = await XboxLiveClient.create(auth_mgr.userinfo.userhash, auth_mgr.xsts_token.jwt, auth_mgr.userinfo.xuid)
    profile = await client.profile.get_profile_by_xuid("2669321029139235")
    profile = await profile.json()
    print(profile)

    achievements = await client.achievements.get_achievements_xboxone_gameprogress("2669321029139235", 219630713)
    achievements = await achievements.json()
    print(achievements)

    await client.close()  # required to properly close aiohttp.ClientSession

asyncio.run(main())

[tuxuser]

This looks good and I would be happy to merge it once completed.

For replacing betamax, this looks like a worthy successor: https://pypi.org/project/vcrpy/

UPDATE: Started on getting betamax replaced by vcrpy:

branch: task/replace_betamax_with_vcrpy

Use the converter like this

$ for cassette_json in tests/data/cassettes/*; do python -m tests.cassette_betamax_to_vcrpy $cassette_json; done

Then test it:

$ pytest

Maybe there needs to be a custom matcher written...

Example output for a converted cassette:

________________________________________________________ test_screenshots_saved_own_titleid_filter _________________________________________________________

vcr_session = <vcr.config.VCR object at 0x10caab3d0>, xbl_client = <xbox.webapi.api.client.XboxLiveClient object at 0x10d62b580>

    def test_screenshots_saved_own_titleid_filter(vcr_session, xbl_client):
        with vcr_session.use_cassette('screenshots_saved_own_titleid.json'):
>           ret = xbl_client.screenshots.get_saved_own_screenshots(title_id=219630713, skip_items=0, max_items=25)

tests/test_screenshots.py:83:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
xbox/webapi/api/provider/screenshots.py:111: in get_saved_own_screenshots
    return self.client.session.get(url, params=params, headers=self.HEADERS_SCREENSHOTS_METADATA)
../../../pyvenv/smartglass/lib/python3.8/site-packages/requests/sessions.py:546: in get
    return self.request('GET', url, **kwargs)
../../../pyvenv/smartglass/lib/python3.8/site-packages/requests/sessions.py:533: in request
    resp = self.send(prep, **send_kwargs)
../../../pyvenv/smartglass/lib/python3.8/site-packages/requests/sessions.py:646: in send
    r = adapter.send(request, **kwargs)
../../../pyvenv/smartglass/lib/python3.8/site-packages/requests/adapters.py:439: in send
    resp = conn.urlopen(
../../../pyvenv/smartglass/lib/python3.8/site-packages/urllib3/connectionpool.py:665: in urlopen
    httplib_response = self._make_request(
../../../pyvenv/smartglass/lib/python3.8/site-packages/urllib3/connectionpool.py:412: in _make_request
    httplib_response = conn.getresponse(buffering=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <vcr.patch.VCRRequestsHTTPSConnection/Users/e99/projects/smartglass/xbox-webapi-python/tests/data/cassettes/screenshots_saved_own_titleid.json object at 0x10cbe99a0>
_ = False, kwargs = {'buffering': True}

    def getresponse(self, _=False, **kwargs):
        """Retrieve the response"""
        # Check to see if the cassette has a response for this request. If so,
        # then return it
        if self.cassette.can_play_response_for(self._vcr_request):
            log.info("Playing response for {} from cassette".format(self._vcr_request))
            response = self.cassette.play_response(self._vcr_request)
            return VCRHTTPResponse(response)
        else:
            if self.cassette.write_protected and self.cassette.filter_request(self._vcr_request):
>               raise CannotOverwriteExistingCassetteException(
                    cassette=self.cassette, failed_request=self._vcr_request
                )
E               vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('/Users/e99/projects/smartglass/xbox-webapi-python/tests/data/cassettes/screenshots_saved_own_titleid.json') in your current record mode ('none').
E               No match for the request (<Request (GET) https://screenshotsmetadata.xboxlive.com/users/me/titles/219630713/screenshots/saved?skipItems=0&maxItems=25>) was found.
E               Found 1 similar requests with 1 different matcher(s) :
E
E               1 - (<Request (GET) https://screenshotsmetadata.xboxlive.com/users/me/titles/219630713/screenshots/saved?maxItems=25&skipItems=0>).
E               Matchers succeeded : ['method']
E               Matchers failed :
E               uri - assertion failure :
E               https://screenshotsmetadata.xboxlive.com/users/me/titles/219630713/screenshots/saved?skipItems=0&maxItems=25 != https://screenshotsmetadata.xboxlive.com/users/me/titles/219630713/screenshots/saved?maxItems=25&skipItems=0

../../../pyvenv/smartglass/lib/python3.8/site-packages/vcr/stubs/__init__.py:231: CannotOverwriteExistingCassetteException

Another example:

________________________________________________________________ test_titlehub_titlehistory ________________________________________________________________

vcr_session = <vcr.config.VCR object at 0x10caab3d0>, xbl_client = <xbox.webapi.api.client.XboxLiveClient object at 0x10d62b580>

    def test_titlehub_titlehistory(vcr_session, xbl_client):
        with vcr_session.use_cassette('titlehub_titlehistory.json'):
            ret = xbl_client.titlehub.get_title_history(987654321)

            assert ret.status_code == 200
>           data = ret.json()

tests/test_titlehub.py:6:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../pyvenv/smartglass/lib/python3.8/site-packages/requests/models.py:897: in json
    return complexjson.loads(self.text, **kwargs)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py:357: in loads
    return _default_decoder.decode(s)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py:337: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <json.decoder.JSONDecoder object at 0x10ba22c70>, s = '', idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.

        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.

        """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py:355: JSONDecodeError

[kyriog]

Hey @tuxuser, thanks for your response and sorry for my inactivity about this.

I’ve started to adapt the unit tests to my aiohttp dev, and I am able to fix the "No match for the request" error by replacing %2C by , (a comma) in converted cassettes.

I’ll finish converting unit testing to aiohttp then I’ll add the commits to this PR.

[tuxuser]

Perfect, thanks!

[tuxuser]

Closing in favor of #29