'NoneType' object has no attribute 'group'

BeatTheBookie commented 1 year ago

Since today or yesterday it's no longer possible to get data from understat.com. The package throws NoneType errors. For me it looks like the website changed the structure a bit and the scraping is no longer working.

Here's the code example I used with Juypter notebooks:

import asyncio import json import pandas as pd import aiohttp

from understat import Understat

lst_leagues = [ "bundesliga" ,"la_liga" ,"serie_a" ,"ligue_1" ,"rfpl" ,"EPL" ]

lst_seasons = [ {"param_season" : "2020", "directory_season" : "2020_21"} ]

async def load_fixtures(league, season): async with aiohttp.ClientSession() as session:

    understat = Understat(session)

    fixtures = await understat.get_league_results(
                league,
                season["param_season"]
            )

    df_fixtures = pd.DataFrame(fixtures)

    await session.close()

    return df_fixtures

for league in lst_leagues:

for season in lst_seasons:

    df_fixtures = await load_fixtures(league, season)

BeatTheBookie commented 1 year ago

Error occurs while decoding the data. Checking whether RegEx still filters the data correct.

amosbastian commented 1 year ago

Others on the Discord server also encountered this problem and apparently solved it with this:

I don't really have time to look into this deeper, but if you can find a fix and want to contribute, feel free to make a PR. Otherwise you can try the workaround (although I don't really understand it 😅).

Nick-Hope commented 1 year ago

Yep, adding cookies={'beget': 'begetok'} seems to fix it. Hoping they don't start changing the names of the cookies!

BeatTheBookie commented 1 year ago

Great, thanks for the feedback.

Let's keep the issue open. I will test, whether the problem can be fixed without the cookie names.

ChrisMusson commented 1 year ago

When making any request to understat.com even in a browser, the beget cookie is passed, so I don't think there is any possibility of a fix without using this.

All tests pass when adding cookies={'beget': 'begetok'} to the fetch function in utils. I think this would be a good fix for now at least.

If anyone wants to create that PR then feel free, or I can do it later this evening.

amosbastian / understat

'NoneType' object has no attribute 'group' #29