search

taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 581 forks source link

Cant scrape tweets #316

Closed leon1995 closed 3 years ago

leon1995 commented 4 years ago

When I try to get tweets the following happens:

print(twitterscraper.query_tweets_from_user("elonmusk", 10))
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 77.73.69.56:80
INFO: Retrying... (Attempts left: 50)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 72.175.68.179:80
INFO: Retrying... (Attempts left: 49)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 34.94.110.134:80
INFO: Retrying... (Attempts left: 48)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 115.144.138.178:7804
INFO: Retrying... (Attempts left: 47)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 103.109.56.202:80
INFO: Retrying... (Attempts left: 46)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 20.44.193.208:80
INFO: Retrying... (Attempts left: 45)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 200.255.122.170:8080
INFO: Retrying... (Attempts left: 44)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 203.202.245.62:80
INFO: Retrying... (Attempts left: 43)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 103.28.121.58:3128
INFO: Retrying... (Attempts left: 42)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 34.208.151.217:3838
INFO: Retrying... (Attempts left: 41)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 52.191.103.11:3128
INFO: Retrying... (Attempts left: 40)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 88.198.24.108:8080
INFO: Retrying... (Attempts left: 39)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 161.35.122.145:8080
INFO: Retrying... (Attempts left: 38)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 181.118.167.104:80
INFO: Retrying... (Attempts left: 37)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 132.255.92.34:53281
INFO: Retrying... (Attempts left: 36)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 209.41.69.101:3128
INFO: Retrying... (Attempts left: 35)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 20.43.156.109:80
INFO: Retrying... (Attempts left: 34)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 138.68.240.218:8080
INFO: Retrying... (Attempts left: 33)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 03.9.34.151:3128
INFO: Retrying... (Attempts left: 32)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 88.99.10.248:1080
INFO: Retrying... (Attempts left: 31)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 162.14.18.11:80
INFO: Retrying... (Attempts left: 30)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 176.9.119.170:8080
INFO: Retrying... (Attempts left: 29)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 51.79.173.166:3128
INFO: Retrying... (Attempts left: 28)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 161.202.226.195:8123
INFO: Retrying... (Attempts left: 27)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 161.202.226.194:8123
INFO: Retrying... (Attempts left: 26)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 14.99.23.26:80
INFO: Retrying... (Attempts left: 25)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 187.17.19.90:44985
INFO: Retrying... (Attempts left: 24)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 193.219.169.234:80
INFO: Retrying... (Attempts left: 23)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 51.68.228.131:80
INFO: Retrying... (Attempts left: 22)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 42.3.51.114:80
INFO: Retrying... (Attempts left: 21)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 35.198.31.85:3128
INFO: Retrying... (Attempts left: 20)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 125.167.100.193:80
INFO: Retrying... (Attempts left: 19)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 80.48.119.28:8080
INFO: Retrying... (Attempts left: 18)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 88.99.10.250:1080
INFO: Retrying... (Attempts left: 17)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 104.41.6.224:80
INFO: Retrying... (Attempts left: 16)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 167.172.152.115:3128
INFO: Retrying... (Attempts left: 15)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 34.208.91.142:3838
INFO: Retrying... (Attempts left: 14)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 27.74.254.12:46401
INFO: Retrying... (Attempts left: 13)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 148.72.172.31:8080
INFO: Retrying... (Attempts left: 12)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 47.91.240.187:80
INFO: Retrying... (Attempts left: 11)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 13.85.25.59:8080
INFO: Retrying... (Attempts left: 10)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 187.130.139.197:8080
INFO: Retrying... (Attempts left: 9)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 34.91.135.38:80
INFO: Retrying... (Attempts left: 8)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 187.73.190.117:8080
INFO: Retrying... (Attempts left: 7)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 117.103.205.244:80
INFO: Retrying... (Attempts left: 6)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 51.79.56.11:8080
INFO: Retrying... (Attempts left: 5)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 103.141.64.250:8191
INFO: Retrying... (Attempts left: 4)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 172.104.57.209:666
INFO: Retrying... (Attempts left: 3)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 3.14.120.193:3838
INFO: Retrying... (Attempts left: 2)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 114.7.163.46:8080
INFO: Retrying... (Attempts left: 1)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 154.127.36.138:8080
INFO: Got 0 tweets from username elonmusk
[]

Hope that someone can help me :)

saliudev commented 4 years ago

See this issue:https://github.com/taspinar/twitterscraper/issues/296

Indeed, this can be fixed by modifying the header dictionary in query.py from HEADER = {'User-Agent': random.choice(HEADERS_LIST)} to HEADER = {'User-Agent': random.choice(HEADERS_LIST), 'X-Requested-With': 'XMLHttpRequest'} that should fix the issue.

AmanSavaria1402 commented 4 years ago

This fixed my issue as well, atleast on windows 10.

sukotyas commented 4 years ago

See this issue:#296

Indeed, this can be fixed by modifying the header dictionary in query.py from HEADER = {'User-Agent': random.choice(HEADERS_LIST)} to HEADER = {'User-Agent': random.choice(HEADERS_LIST), 'X-Requested-With': 'XMLHttpRequest'} that should fix the issue.

this fixed my issue.. thanks..

AveryData commented 4 years ago

this worked for me too....should it be changed in the git repo here?

taspinar commented 3 years ago

This fix has been merge to twitterscraper and available in version 1.5.0 on pypi as well.