search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.75k stars 2.72k forks source link

you sure this is working? #1095

Open azamet90 opened 3 years ago

azamet90 commented 3 years ago

twint -u rihanna
CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)

rach-sharp commented 3 years ago

also getting this issue, stepped through the code and found that it happens due to

<h1>This browser is no longer supported.</h1>
<p>
      Please switch to a supported browser to continue using twitter.com. You can see a list of supported browsers in our Help Center.
    </p>
<p class="errorButton"><a href="https://help.twitter.com/using-twitter/twitter-supported-browsers">Help Center</a>
</p>

and then BeautifulSoup fails to find a <div> with class="user-actions"

rach-sharp commented 3 years ago

the call to get.User wasn't actually specifying a User-Agent. I fiddled around with the code so that it did, and now it's blocked by a lack of Javascript. This is using one of the list of User-Agents that the lib chooses randomly for some twitter endpoints when scraping.

Suggests that it needs a Selenium-style browser to scrape Twitter now or some other way to load Javascript

<h1>JavaScript is not available.</h1>
      <p>We've detected that JavaScript is disabled in this browser. Please enable JavaScript or switch to a supported
        browser to continue using twitter.com. You can see a list of supported browsers in
        our Help Center.</p>
      <p class="errorButton"><a href="https://help.twitter.com/using-twitter/twitter-supported-browsers">Help Center</a>
      </p>
Altimis commented 3 years ago

@rach-sharp @azamet90 check this

i-infra commented 3 years ago

Syndication timelines can be used to scrape up to 3000 likes. The new twitter API doesn't support the previous (mobile web) route to view content. Timeline etc still works fine.

vishwesh-D-kumar commented 3 years ago

@i-infra how do i use the syndication timelines to scrape likes ? I'm new to twint. A small sample usage might be of great help , if possible.