shaikhsajid1111 / twitter-scraper-selenium

Python's package to scrap Twitter's front-end easily
https://pypi.org/project/twitter-scraper-selenium
MIT License
322 stars 50 forks source link

Scraping only 4 tweets no matter the tweet count #49

Open Orski174 opened 2 years ago

Orski174 commented 2 years ago

As mentioned in the title, I'm getting only 4 tweets using this code

from twitter_scraper_selenium import scrape_profile
import os
import json
account = input("Account: ")
tweets = int(input("How many tweets: "))
path = os.path.join(parent_dir, account)
if os.path.exists(path) == False:
    os.mkdir(path)
data = scrape_profile(twitter_username=account, output_format="json", browser="firefox", tweets_count=tweets)
print(data)
parsed = json.loads(data)
json_data = json.dumps(parsed, indent=4)
with open(path + "\\" + account + ".json", "w") as outfile:
    outfile.write(json_data)

And this is the print output: https://pastebin.com/p1UuxZFa

Thank you.

shaikhsajid1111 commented 2 years ago

I see, I think there might be some sort of update on front end side of the Twitter. I'm currently unavailable for pushing out fix so I would suggest to use scrape_keyword_with_api function with query built from here which will filter out tweets specifically posted by the given username. Example query could be something like from:BarackObama