1220moritz / reverse-twitter-scraper

A request based package for scraping twitter data. No API Key required. Support for proxies and private twitter accounts
Discord: moritz#1220
MIT License
26 stars 3 forks source link

TypeError: WebDriver.__init__() got multiple values for argument 'options' #2

Closed 0xking-sol closed 12 months ago

0xking-sol commented 1 year ago

Ran into an error when trying to run the code. Might be a version compatibilty issue, sharing details here.

Approach:

  1. Copied entire cookies line and pasted into cookies = "Cookies_Pasted_Here".
Screenshot 2023-07-04 at 11 42 41
  1. Tried to run the example script:
    
    from ReverseTwitterScraper import TwitterScraper

chromedriver_path = "C:/Program Files (x86)/chromedriver.exe" cookies = "" #no account cookies cookies = {'Cookie': 'your Cookie', 'X-Csrf-Token': 'your csrf token'} #with account cookie proxy_list = []

twitter_handle = ["elonmusk"] # single account twitter_handles = ["elonmusk", "POTUS", "latestinspace"] # multiple accounts scraper = TwitterScraper(twitter_handle, chromedriver_path, cookies, proxy_list) tweets = scraper.getTweetsText()

print(tweets)



3. Following errors:
TypeError: Invalid type for url.  Expected str or httpx.URL, got <class 'NoneType'>: None
TypeError: WebDriver.__init__() got multiple values for argument 'options'

### Troubleshooting:
1. Chromedriver works by itself with other scripts
3. Selenium version 4.10.0 
4. Macbook air M1, using pycharm 

Could you also please help me understand the distinction between cookie and X-Csrf-Token in the code?

Thanks!
1220moritz commented 1 year ago

The account cookie and X-Csrf-Token have to match. You need to get both when you manually copy the cookie. This is not the issue though. Twitter requires login now to view/scrape tweets. Even if you want to use account cookies with my scraper, it tries to get the auth cookies before and just sends the "getTweets" requests with account cookies. I think it's currently broken and I'm trying to fix it asap.

1220moritz commented 12 months ago

0.8 mandates the use of account cookies for viewing tweets. "Fixes" the problem

0xking-sol commented 12 months ago

Thanks for continuously working on this. 0.8 refers to your code version?

I also wrote a simple script for exporting a user's own bookmarks which I'd be happy to add as functionality here if that would be of interest

1220moritz commented 12 months ago

0.8 indeed does refer to the code version. As Twitter only allows logged-in accounts to view tweets, the use of cookies is mandatory. This comes with a high risk of an account ban. I found a way to scrape tweets without the need for an account but for all other functions, one is still needed. (not included yet) If you want to add your functionality, you can open a pull request, which will be merged after review.