search

godkingjay / selenium-twitter-scraper

This is a Twitter Scraper which uses Selenium for scraping tweets. It is capable of scraping tweets from home, user profile, hashtag, query or search, and advanced searches.
Apache License 2.0
177 stars 47 forks source link

(request) Scrape each poster's [FOLLOWERS] and [FOLLOWING] count? #4

Closed badjoerichards closed 1 year ago

badjoerichards commented 1 year ago

Is it possible to get the user's [FOLLOWERS] and [FOLLOWING] count as 2 new columns of data in the .csv?

Very useful information when scraping.

Thanks

godkingjay commented 1 year ago

It's possible to get the [FOLLOWING] and [FOLLOWERS] count of the tweet's poster, however this will definitely slow down the scraping process because in twitter, the user's followers and following is not directly included in the tweet. So I will only add them as an optional data to be scraped. I'll make a commit later, I'm currently doing some tests and optimizing the scraper. There will be significant improvements to the scraper especially when scraping without including the [FOLLOWERS] and [FOLLOWING] count.

godkingjay commented 1 year ago

To scrape followers and following count:

python scraper --add="pd"

feat: optionally scrape followers and following

docs: Sample code for additional data to scrape.

badjoerichards commented 1 year ago

This is great! Thank you