search

ptwobrussell / Mining-the-Social-Web

The official online compendium for Mining the Social Web (O'Reilly, 2011)
http://bit.ly/135dHfs
Other
1.21k stars 490 forks source link

Problems in chapter5 with ImportError: No module named twitter__util #73

Open SamJingZhao opened 10 years ago

SamJingZhao commented 10 years ago

Example 5-3. Harvesting tweets from a user or public timeline (the_tweet__harvest_timeline.py)

import sys
import time
import twitter
import couchdb
import sys
print  sys.path
sys.path.append('/Users/hike/Documents/Mining-the-Social-Web-master/python_code/twitter__util.py')
import twitter__util
from couchdb.design import ViewDefinition
from twitter__util import makeTwitterRequest
from twitter__util import getNextQueryMaxIdParam

# Go to http://twitter.com/apps/new to create an app and get these items
# See https://dev.twitter.com/docs/auth/oauth for more information on Twitter's OAuth implementation
def oauth_login():
    CONSUMER_KEY = ''
    CONSUMER_SECRET = ''
    OAUTH_TOKEN = ''
    OAUTH_TOKEN_SECRET = ''
    auth = twitter.oauth.OAuth(OAUTH_TOKEN, OAUTH_TOKEN_SECRET,
                           CONSUMER_KEY, CONSUMER_SECRET)

twitter_api = oauth_login()

TIMELINE_NAME = 'user' # XXX: IPython Notebook cannot prompt for input
MAX_PAGES = 2 # XXX: IPython Notebook cannot prompt for input
USER = 'acarvin' # XXX: IPython Notebook cannot prompt for input

KW = {  # For the Twitter API call
    'count': 200,
    'trim_user': 'true',
    'include_rts' : 'true',
    'since_id' : 1,
    }

if TIMELINE_NAME == 'user':
    USER = sys.argv[3]
    KW['screen_name'] = USER
if TIMELINE_NAME == 'home' and MAX_PAGES > 4:
    MAX_PAGES = 4
if TIMELINE_NAME == 'user' and MAX_PAGES > 16:
    MAX_PAGES = 16

t = login()

# Establish a connection to a CouchDB database
server = couchdb.Server('http://localhost:5984')
DB = 'tweets-%s-timeline' % (TIMELINE_NAME, )

if USER:
    DB = '%s-%s' % (DB, USER)

try:
    db = server.create(DB)
except couchdb.http.PreconditionFailed, e:

    # Already exists, so append to it, keeping in mind that duplicates could occur

    db = server[DB]

    # Try to avoid appending duplicate data into the system by only retrieving tweets 
    # newer than the ones already in the system. A trivial mapper/reducer combination 
    # allows us to pull out the max tweet id which guards against duplicates for the 
    # home and user timelines. This is best practice for the Twitter v1.1 API
    # See https://dev.twitter.com/docs/working-with-timelines

    def idMapper(doc):
        yield (None, doc['id'])

    def maxFindingReducer(keys, values, rereduce):
        return max(values)

    view = ViewDefinition('index', 'max_tweet_id', idMapper, maxFindingReducer,
                          language='python')
    view.sync(db)

    KW['since_id'] = int([_id for _id in db.view('index/max_tweet_id')][0].value)

api_call = getattr(t.statuses, TIMELINE_NAME + '_timeline')
tweets = makeTwitterRequest(api_call, **KW)
db.update(tweets, all_or_nothing=True)
print 'Fetched %i tweets' % len(tweets)

page_num = 1
while page_num < MAX_PAGES and len(tweets) > 0:

    # Necessary for traversing the timeline in Twitter's v1.1 API.
    # See https://dev.twitter.com/docs/working-with-timelines
    KW['max_id'] = getNextQueryMaxIdParam(tweets)

    api_call = getattr(t.statuses, TIMELINE_NAME + '_timeline')
    tweets = makeTwitterRequest(api_call, **KW)
    db.update(tweets, all_or_nothing=True)
    print 'Fetched %i tweets' % len(tweets)
    page_num += 1

ImportError Traceback (most recent call last)

in () 6 print sys.path 7 sys.path.append('/Users/hike/Documents/Mining-the-Social-Web-master/python_code/twitter__util.py') ----> 8 import twitter__util 9 from couchdb.design import ViewDefinition 10 from twitter__util import makeTwitterRequest ImportError: No module named twitter__util ['', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/virtualenv-1.10.1-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ipython-1.0.0-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.3.0-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyzmq-13.1.0-py2.7-macosx-10.6-intel.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Pygments-1.6-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Sphinx-1.2b1-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tornado-3.1.1-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MarkupSafe-0.18-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/docutils-0.11-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-1.1.4-py2.7.egg', '/Library/Python/2.7/site-packages/readline-6.2.4.1-py2.7-macosx-10.7-intel.egg', '/Library/Python/2.7/site-packages/Jinja2-2.7.1-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twitter-1.10.0-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyYAML-3.10-py2.7-macosx-10.6-intel.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twitter_text_py-2.0.0-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/prettytable-0.7.2-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/networkx-1.8.1-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/CouchDB-0.9-py2.7.egg', '/Library/Python/2.7/site-packages/ipython-1.0.0_dev-py2.7.egg', '/Library/Python/2.7/site-packages/matplotlib-1.4.x-py2.7-macosx-10.8-intel.egg', '/Library/Python/2.7/site-packages/numpy-1.8.0.dev_928289b_20130808-py2.7-macosx-10.8-intel.egg', '/Library/Python/2.7/site-packages/pandas-0.12.0_146_g9cba0de_20130808-py2.7-macosx-10.8-intel.egg', '/Library/Python/2.7/site-packages/pymc-2.2-py2.7-macosx-10.8-intel.egg', '/Library/Python/2.7/site-packages/scikit_learn-0.14_git-py2.7-macosx-10.8-intel.egg', '/Library/Python/2.7/site-packages/scipy-0.13.0.dev_c31f167_20130808-py2.7-macosx-10.8-intel.egg', '/Library/Python/2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-intel.egg', '/Library/Python/2.7/site-packages/nose-1.3.0-py2.7.egg', '/Library/Python/2.7/site-packages/six-1.4.1-py2.7.egg', '/Library/Python/2.7/site-packages/pyparsing-1.5.7-py2.7.egg', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/Library/Python/2.7/site-packages/pytz-2013b-py2.7.egg', '/Library/Python/2.7/site-packages/tornado-3.1.1-py2.7.egg', '/Library/Python/2.7/site-packages/pyzmq-13.1.0-py2.7-macosx-10.6-intel.egg', '/Library/Python/2.7/site-packages/pika-0.9.13-py2.7.egg', '/Library/Python/2.7/site-packages/MarkupSafe-0.18-py2.7.egg', '/Library/Python/2.7/site-packages/patsy-0.2.1-py2.7.egg', '/Library/Python/2.7/site-packages/Pygments-1.6-py2.7.egg', '/Library/Python/2.7/site-packages/Sphinx-1.2b1-py2.7.egg', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages', '/Library/Python/2.7/site-packages', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ipython-1.0.0-py2.7.egg/IPython/extensions', 'documents:\Mining-the-Social-Web-master\python_code\twitter__util.py', 'documents:\Mining-the-Social-Web-master\python_code\twitter__util.py', '/Users/hike/Documents/Mining-the-Social-Web-master/python_code/twitter__util.py', '/Users/hike/Documents/Mining-the-Social-Web-master/python_code/twitter__util.py', '/Users/hike/Documents/Mining-the-Social-Web-master/python_code/twitter__util.py', '/Users/hike/Documents/Mining-the-Social-Web-master/python_code/twitter__util.py']
ptwobrussell commented 10 years ago

Hi - In the source code checkout, twitterutil.py is in the same directory as the_tweetharvest_timeline.py file, and as long as you are trying to run the script the_tweetharvcest_timeline.py from its working directory, everything should work fine. e.g. in a terminal, you would navigate to the python_code directory and then type "python the_tweetharvest_timeline.py" along with additional arguments.

The only reason you should get that import error is if you are trying to run the code from some other directory. Can you give that a shot (and provide more detail about how you are trying to run the script) if that doesn't help?

I'd also highly recommend watching these videos - https://vimeo.com/channels/MiningTheSocialWeb - which show you how to get started with the 2nd Edition virtual machine, which provides a much better/simpler coding experience in comparison if that's something you're willing to consider. The 2nd Edition repo is here if you want to take a look - https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition

SamJingZhao commented 10 years ago

Dear Matthew, I just import your document of Chapter 5 into iPython notebook, so do that means it in the other directory? I really love the 2nd Edition, why I am still learning the 1st Edition as many great examples in Chapter 5 of 1st Edition are not in the 2nd Edition. I will go to watch the videos, many thanks, I appreciate your support!

ptwobrussell commented 10 years ago

@SamJingZhao - Check out Chapter 9 of the 2nd Edition, which is linked off of its main GitHub repo page as a viewable notebook here - https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/master/ipynb/html/Chapter%209%20-%20Twitter%20Cookbook.html

I think that the majority of the "good stuff" from Chapter 5 in the 1E is in that notebook in a much more consumable fashion, so you might have some good luck with it.

So it sounds like you are running IPython Notebook with the 1E code? If this is the case, then all you should have to do if you are working in IPython Notebook is have the ancillary files like twitter__util.py in the same directory that your Chapter 5 IPython notebook file is at so that you can execute it.

Alternatively (but a less preferred method) is that you could just open up files like twitter__util.py and copy/paste in the code into a notebook cell, execute it, and then it would be as though you have those functions available to you, and you wouldn't even need to import them.

Does that all make sense?

Let me know if this works for you. I want to make sure that you're taken care of and happy. I may have to sign off till the morning pretty soon, but I will check back on you in the AM. If you don't get this sorted by them, I will create you a custom Chapter 5 notebook with everything in it that you need so that you don't have to deal with the import errors, but try my advice in the meanwhile and see if it helps.

SamJingZhao commented 10 years ago

Dear Matthew, Sorry for my late reply, over night's of learning this book, I am almost sick and didn't reply you last night. Which example in Chapter 5 I need and I didn't found in the 2nd Edition is Example 5-12. Finding the tweets that have been retweeted most often (the_tweet_count_retweets_by_others.py)

I am hoping to using these example for analyzing how tweets impact world democracy. Could you give me suggestions or help me for these coding in the new API system? Many thanks. Jing

ptwobrussell commented 10 years ago

I believe that all of the examples that you're interested in are in the Twitter Cookbook that you can find here - https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/master/ipynb/html/Chapter%209%20-%20Twitter%20Cookbook.html

Take a look at that and let me know if you think it has what you're looking for. The recipes are designed to be pretty easy to compose (sort of like lego building blocks) so if you are already doing a little bit of programming in Python, I think you'll be able to put them together pretty easily. Let me know, though, if this isn't what you're looking for.

You may also find this wiki page generally helpful that provides a full catalog of all numbered examples from the 2nd Edition - https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/wiki/Numbered-Examples