search

bisguzar / twitter-scraper

Scrape the Twitter Frontend API without authentication.
MIT License
3.88k stars 601 forks source link

[refactor] fix Profile object #167

Open yimingStar opened 3 years ago

yimingStar commented 3 years ago

Uable to get the original page (400 error code), get mobile twitter page indead.

  1. Change the User-Agent
  2. Fix the selector queries (Missing getting birthday, favorite and is_verified)

    166

yeachan153 commented 3 years ago

Hey, just passing and I don't really know the code enough to do a PR. Just to let you know I tried your branch and it did not work for me, although the issue is not related with the user agent I think.

image

yimingStar commented 3 years ago

@yeachan153 Genuinely agree with your suggestion for the purpose of letting the user notice the missing web elements. So I added the exception function into this Profile class, which will print out the missing attributes.

The fixed is in this commit https://github.com/bisguzar/twitter-scraper/pull/167/commits/c7bbf55f5bfe94255242035f82237123f4581c38

HJoentgen commented 3 years ago

Is this solution still working? When I use the new profile.py, I still get the same "local variable 'html' referenced before assignment" error, because of a 400 response.

So this:

from twitter_scraper import Profile
profile = Profile(f"BarackObama")
print(profile.to_dict())

leads to:

Unable to get is_private in html, exception - local variable 'html' referenced before assignment
...