search

bisguzar / twitter-scraper

Scrape the Twitter Frontend API without authentication.
MIT License
3.88k stars 601 forks source link

UnboundLocalError: local variable 'html' referenced before assignment #166

Open jalilbm opened 3 years ago

jalilbm commented 3 years ago

I tried a basic profile selection:

profile = Profile(user_name)

And I got this error:

File "/usr/local/lib/python3.6/site-packages/twitter_scraper/modules/profile.py", line 65, in __parse_profile
    self.location = html.find(".ProfileHeaderCard-locationText")[0].text
UnboundLocalError: local variable 'html' referenced before assignment

I believe that something changed on twitter website

pradeepbatchu commented 3 years ago

I also got same issue, below is the error 

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): twitter.com:443
DEBUG:urllib3.connectionpool:https://twitter.com:443 "GET /pradeepbatchu HTTP/1.1" 400 None
Traceback (most recent call last):
  File "D:/twitt.py", line 15, in <module>
    gettwitter_profile('pradeepbatchu')
  File "D:/twitt.py", line 7, in gettwitter_profile
    profile = Profile(twitter_handler)
  File "C:\Users\prade\anaconda3\lib\site-packages\twitter_scraper\modules\profile.py", line 41, in __init__
    self.__parse_profile(page)
  File "C:\Users\prade\anaconda3\lib\site-packages\twitter_scraper\modules\profile.py", line 65, in __parse_profile
    self.location = html.find(".ProfileHeaderCard-locationText")[0].text
UnboundLocalError: local variable 'html' referenced before assignment
LeiLuo2020 commented 3 years ago

I also got the same error to what jalilrooney reported.

yimingStar commented 3 years ago

By getting the response code <400> from the request; it seems like the old version server side render api has been changed. I just get the html by replacing the User-Agent. However, the selectors query need to change, since the new webpage did not contain the same column.

danielp3011 commented 3 years ago

I have the same error. Can somebody point out a way to solve it. Sadly I'm not an experienced cooder yet. How did you solve this with the User-Agent @yimingStar? Can you explain that in detail? That would be nice! Thanks a lot

yimingStar commented 3 years ago

Hi, @danielp3011 I made a pull request here - https://github.com/bisguzar/twitter-scraper/pull/167

Because the original page cannot show up, I use another user-agent to get a mobile version web page (mobile.twitter.com). Next, we'll have to rewrite the selectors, since the class name of the web elements have changed. By the way, there still are some elements we cannot get, but most of them are fine.

bisguzar commented 3 years ago

Please leave a review for PR @danielp3011 . So I can merge it easyly. Thanks for your efforts guys!

J0ANMM commented 3 years ago

Should this fix still be working? I tried it on my local file but I am still getting the 'html' error...

rinaldipratama commented 3 years ago

Same, I still got the error. "UnboundLocalError: local variable 'html' referenced before assignment."

Bitcoinera commented 1 year ago

getting the same error here. Is there no workaround?

mominurr commented 9 months ago

I am getting same error.