search

taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 581 forks source link

readme

|Downloads| |Downloads_month| |PyPI version| |GitHub contributors|

.. |Downloads| image:: https://pepy.tech/badge/twitterscraper :target: https://pepy.tech/project/twitterscraper .. |Downloads_month| image:: https://pepy.tech/badge/twitterscraper/month :target: https://pepy.tech/project/twitterscraper/month .. |PyPI version| image:: https://badge.fury.io/py/twitterscraper.svg :target: https://badge.fury.io/py/twitterscraper .. |GitHub contributors| image:: https://img.shields.io/github/contributors/taspinar/twitterscraper.svg :target: https://github.com/taspinar/twitterscraper/graphs/contributors

Backers

Thank you to all our backers! 🙏 [Become a backer_]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor_]

.. _Become a backer: https://opencollective.com/twitterscraper#backer .. _Become a sponsor: https://opencollective.com/twitterscraper#sponsor

Synopsis

A simple script to scrape Tweets using the Python package requests to retrieve the content and Beautifulsoup4 to parse the retrieved content.

  1. Motivation

Twitter has provided REST API's <https://dev.twitter.com/rest/public> which can be used by developers to access and read Twitter data. They have also provided a Streaming API <https://dev.twitter.com/streaming/overview> which can be used to access Twitter Data in real-time.

Most of the software written to access Twitter data provide a library which functions as a wrapper around Twitter's Search and Streaming API's and are therefore constrained by the limitations of the API's.

With Twitter's Search API you can only send 180 Requests every 15 minutes. With a maximum number of 100 tweets per Request, you can mine 72 tweets per hour (4 x 180 x 100 =72) . By using TwitterScraper you are not limited by this number but by your internet speed/bandwith and the number of instances of TwitterScraper you are willing to start.

One of the bigger disadvantages of the Search API is that you can only access Tweets written in the past 7 days. This is a major bottleneck for anyone looking for older data. With TwitterScraper there is no such limitation.

Per Tweet it scrapes the following information:

In addition it can scrape for the following user information:

  1. Installation and Usage

To install twitterscraper:

.. code:: python

(sudo) pip install twitterscraper

or you can clone the repository and in the folder containing setup.py

.. code:: python

python setup.py install

If you prefer more isolation you can build a docker image

.. code:: python

docker build -t twitterscraper:build .

and run your container with:

.. code:: python

docker run --rm -it -v/<PATH_TO_SOME_SHARED_FOLDER_FOR_RESULTS>:/app/data twitterscraper:build <YOUR_QUERY>

2.2 The CLI

You can use the command line application to get your tweets stored to JSON right away. Twitterscraper takes several arguments:

2.2.1 Examples of simple queries


Below is an example of how twitterscraper can be used:

``twitterscraper Trump --limit 1000 --output=tweets.json``

``twitterscraper Trump -l 1000 -o tweets.json``

``twitterscraper Trump -l 1000 -bd 2017-01-01 -ed 2017-06-01 -o tweets.json``

2.2.2 Examples of advanced queries

You can use any advanced query Twitter supports. An advanced query should be placed within quotes, so that twitterscraper can recognize it as one single query.

Here are some examples:

You can construct an advanced query on Twitter Advanced Search <https://twitter.com/search-advanced?lang=en> or use one of the operators shown on this page <https://lifehacker.com/search-twitter-more-efficiently-with-these-search-opera-1598165519>. Also see Twitter's Standard operators <https://developer.twitter.com/en/docs/tweets/search/guides/standard-operators.html>__

2.2.3 Examples of scraping user pages



You can also scraped all tweets written or retweeted by a specific user.
This can be done by adding the boolean argument ``-u / --user`` argument.
If this argument is used, the search term should be equal to the username.

Here is an example of scraping a specific user:

``twitterscraper realDonaldTrump --user -o tweets_username.json``

This does not work in combination with ``-p``, ``-bd``, or ``-ed``.

The main difference with the example "search for tweets from a specific user" in section 2.2.2 is that this method really scrapes
all tweets from a profile page (including retweets).
The example in 2.2.2 scrapes the results from the search page (excluding retweets).

2.3 From within Python
----------------------

You can easily use TwitterScraper from within python:

::

    from twitterscraper import query_tweets

    if __name__ == '__main__':
        list_of_tweets = query_tweets("Trump OR Clinton", 10)

        #print the retrieved tweets to the screen:
        for tweet in query_tweets("Trump OR Clinton", 10):
            print(tweet)

        #Or save the retrieved tweets to file:
        file = open(“output.txt”,”w”)
        for tweet in query_tweets("Trump OR Clinton", 10):
            file.write(str(tweet.text.encode('utf-8')))
        file.close()

2.3.1 Examples of Python Queries
--------------------------------

   - Query tweets from a given URL:
      Parameters:
         - query:     The query search parameter of url
         - lang:      Language of queried url
         - pos:       Parameter passed for where to start looking in url
         - retry:     Number of times to retry if error   

      .. code:: python

          query_single_page(query, lang, pos, retry=50, from_user=False, timeout=60)

   - Query all tweets that match qeury:
      Parameters:
         - query:     The query search parameter
         - limit:     Number of tweets returned
         - begindate: Start date of query
         - enddate:   End date of query
         - poolsize:  Tweets per poolsize
         - lang:      Language of query

      .. code:: python

          query_tweets('query', limit=None, begindate=dt.date.today(), enddate=dt.date.today(), poolsize=20, lang='')

   - Query tweets from a specific user:
      Parameters:
         - user:      Twitter username
         - limit:     Number of tweets returned

      .. code:: python

          query_tweets(user, limit=None)

2.4 Scraping for retweets
----------------------

A regular search within Twitter will not show you any retweets.
Twitterscraper therefore does not contain any retweets in the output.

To give an example: If user1 has written a tweet containing ``#trump2020`` and user2 has retweetet this tweet,
a search for ``#trump2020`` will only show the original tweet.

The only way you can scrape for retweets is if you scrape for all tweets of a specific user with the ``-u / --user`` argument.

2.5 Scraping for User Profile information
----------------------
By adding the argument ``--profiles`` twitterscraper will in addition to the tweets, also scrape for the profile information of the users who have written these tweets.
The results will be saved in the file "userprofiles_<filename>".

Try not to use this argument too much. If you have already scraped profile information for a set of users, there is no need to do it again :)
It is also possible to scrape for profile information without scraping for tweets.
Examples of this can be found in the examples folder.

3. Output
=========

All of the retrieved Tweets are stored in the indicated output file. The
contents of the output file will look like:

::

    [{"fullname": "Rupert Meehl", "id": "892397793071050752", "likes": "1", "replies": "0", "retweets": "0", "text": "Latest: Trump now at lowest Approval and highest Disapproval ratings yet. Oh, we're winning bigly here ...\n\nhttps://projects.fivethirtyeight.com/trump-approval-ratings/?ex_cid=rrpromo\u00a0\u2026", "timestamp": "2017-08-01T14:53:08", "user": "Rupert_Meehl"}, {"fullname": "Barry Shapiro", "id": "892397794375327744", "likes": "0", "replies": "0", "retweets": "0", "text": "A former GOP Rep quoted this line, which pretty much sums up Donald Trump. https://twitter.com/davidfrum/status/863017301595107329\u00a0\u2026", "timestamp": "2017-08-01T14:53:08", "user": "barryshap"}, (...)
    ]

3.1 Opening the output file
---------------------------

In order to correctly handle all possible characters in the tweets
(think of Japanese or Arabic characters), the output is saved as utf-8
encoded bytes. That is why you could see text like
"\u30b1 \u30f3 \u3055 \u307e \u30fe ..." in the output file.

What you should do is open the file with the proper encoding:

.. figure:: https://user-images.githubusercontent.com/4409108/30702318-f05bc196-9eec-11e7-8234-a07aabec294f.PNG

   Example of output with Japanese characters

3.1.2 Opening into a pandas dataframe
---------------------------

After the file has been opened, it can easily be converted into a ```pandas``` DataFrame

::

    import pandas as pd
    df = pd.read_json('tweets.json', encoding='utf-8')