search

dcstats / CBBpy

A Python-based web scraper for NCAA basketball.
MIT License
11 stars 2 forks source link

Some games print exception "AttributeError: 'NoneType' object has no attribute 'get_text'" #1

Closed dcstats closed 1 year ago

dcstats commented 2 years ago

Exception sometimes printed when scraping season games. Expected behavior is for exceptions to be caught and logged but not printed.

Code

s.get_games_season(2021)

Stacktrace

Scraping game 401261570 (8/10) on 2020-12-09 00:00:00:  21%|▏| 38/181 [02:17<17:
Traceback (most recent call last):
   File "/Users/dscow/Documents/Projects/CBBpy/testing/espn/mens/scrape.py", line 219, in get_game_info
    if bad_game in status_div.get_text():
AttributeError: 'NoneType' object has no attribute 'get_text'
dcstats commented 2 years ago

Additionally, some games get this error logged because the page isn't loaded properly (when using a browser, the page that appears that causes the problem is one that says "Page not found. Go to the homepage.") Using longer duration of time.sleep() could help

dcstats commented 2 years ago

Extra time.sleep() doesn't necessarily solve the issue of page not loading properly. Gotta do more digging

dcstats commented 1 year ago

This should be mostly resolved with latest versions. Some games have inherent issues though and can't be scraped - those are ones that always say 'Page not found' when loading in browser (https://www.espn.com/mens-college-basketball/game/_/gameId/400508100 is an example of this).