scrape-twitter 
🐦 Access Twitter data without an API key
DEPRECATED: The next version of Twitter auto-generates all of the class names and therefore is going to be much more difficult to screen scrape. I might come up with a better solution...
This module provides command line interfaces to scrape: profiles, timelines, connections, likes, search and conversations.
It also exposes both streams and a promise returning function to help accessing Twitter in your own applications.
Real-time firehoses can be created using the companion module monitor-head-stream.
Features
- [x] Get Twitter data without being required to configure an API key.
- [x] Twitter can't constrain access as easily as they can to an API or an individual API key. Any constraints introduced would apply to their public site. A scraper can be fixed; you are no longer beholden to Twitter.
- [x] Grab timelines, whole conversations, profiles, connections, likes, etc.
- [x] Automatically pages to fetch all tweets.
- [x] Provides metadata on how tweet replies are linked together. e.g.
isReplyToId
Example
Get profile
$ scrape-twitter profile sebinsua
# ...Get timeline
$ scrape-twitter timeline nouswaves
# ...Get likes
This command requires a valid login. It will check for the following environment variables: TWITTER_USERNAME, TWITTER_PASSWORD, TWITTER_KDT. But can also pick these up from a dotenv file at the path ~/.scrape-twitter. The first time you login you will be asked to store the TWITTER_KDT - this is used by Twitter to recognise your device.
$ scrape-twitter likes sebinsua
# ...Get connections
This command also requires a valid login.
$ scrape-twitter connections sebinsua --type=following
# ...Get conversation
$ scrape-twitter conversation ctbeiser 691766715835924484
# ...Search
$ scrape-twitter search --query "from:afoolswisdom motivation" --type latest
# ...Get list
$ scrape-twitter list nouswaves list
# ...JSON interface plays nicely with CLI tools like jq, coreutils/gshuf and terminal-notifier
For example, a MOTD-like script might contain:
scrape-twitter search --query="from:afoolswisdom knowledge" | jq -r '.[].text' | gshuf -n 1 | terminal-notifier -title "Knowledge (MOTD)"Install
With yarn:
yarn global add scrape-twitterWith npm:
npm install -g scrape-twitterAPI
new TimelineStream(username: string, { retweets: boolean, replies: boolean, count: ?number })
Create a ReadableStream<Tweet> for the timeline of a username.
new LikeStream(username: string, { count: ?number, env: process.env })
Create a ReadableStream<Tweet> for the likes of a username.
new ConnectionStream(username: string, type: 'following' | 'followers', process.env)
Create a ReadableStream<UserConnection> for the connections of a username.
new ConversationStream(username: string, id: string, { count: ?number })
Create a ReadableStream<Tweet> for the conversation that belongs to a username and tweet id.
new ThreadedConversationStream(id: string)
Create a ReadableStream<Tweet> for the thread that belongs to a tweet id.
new TweetStream(query: string, type: 'top' | 'latest', { count: ?number })
Create a ReadableStream<Tweet> for the tweets that match a query and type.
new ListStream(username: string, list: string, { count: ?number })
Create a ReadableStream<Tweet> for the username's list.
getUserProfile(username: string)
Get a Promise<UserProfile> for a particular username.