fake-name / xA-Scraper

69 stars 8 forks source link

TwitGet: Handle AssertionError if target user is suspended #88

Closed God-damnit-all closed 4 years ago

God-damnit-all commented 4 years ago

If the target user is suspended (which can be temporary or permanent, there's no way of knowing), the log should throw a warning and catch the following AssertionError before moving on to the next target:

Main.TwitGet.StatusMgr - ERROR - Traceback (most recent call last):
Main.TwitGet.StatusMgr - ERROR -   File "D:\xA-Scraper\xascraper\modules\twit\twitScrape.py", line 249, in go
Main.TwitGet.StatusMgr - ERROR -     errored |= self.getArtist(aid=aid, artist=name, ctrlNamespace=ctrlNamespace)
Main.TwitGet.StatusMgr - ERROR -   File "D:\xA-Scraper\xascraper\modules\twit\twitScrape.py", line 199, in getArtist
Main.TwitGet.StatusMgr - ERROR -     for tweet in intf.get_all_tweets(artist, min_date):
Main.TwitGet.StatusMgr - ERROR -   File "D:\xA-Scraper\xascraper\modules\twit\vendored_twitter_scrape.py", line 274, in get_all_tweets
Main.TwitGet.StatusMgr - ERROR -     interval_start = self.get_joined_date(username)
Main.TwitGet.StatusMgr - ERROR -   File "D:\xA-Scraper\xascraper\modules\twit\vendored_twitter_scrape.py", line 148, in get_joined_date
Main.TwitGet.StatusMgr - ERROR -     assert joined_items, "No joined items?"
Main.TwitGet.StatusMgr - ERROR - AssertionError: No joined items?
fake-name commented 4 years ago

You can tell this is well tested lol.

God-damnit-all commented 4 years ago

@fake-name Make sure you look at my PRs before you do anything on your local branches

fake-name commented 4 years ago

Yeah, will do.

fake-name commented 4 years ago

Merged those PRs. Thanks a bunch! I think you're testing the twitter stuff a heck of a lot more thoroughly then I have so far.

fake-name commented 4 years ago

Best thing to do here is actually throw a AccountDisabledException() which kicks scraping the artist to the end of the to-fetch queue.