twitter_search with geocode

[vlaco]

Hi,

I am using the code from the book to use twitter_search and I am trying to use slightly more complex query for my project. The code is running when I use a simple query such as q = 'Migrants' But whenever I try to use q = "migrants%2C%20OR%20migrations%2C%20OR%20refugees%20&geocode=%2244.802416%2C20.465601%2C80mi%22" I get this error print json.dumps(results[0], indent=1) IndexError: list index out of range which I am guessing it means that it is not fining any tweets under these conditions but when I use the same query in Exploring the Twitter API console it is working.

Any ideas what could be wrong please?

[ptwobrussell]

Is your query including the URL encoding exactly as shown with %2C%20, etc.? If so, I think you should try it without the url encoding. e.g. "migrants OR migrations OR refugees"...

On Thu, Sep 17, 2015 at 7:46 PM, Vanja Vlaco notifications@github.com wrote:

Hi,

I am using the code from the book to use twitter_search and I am trying to use slightly more complex query for my project. The code is running when I use a simple query such as q = 'Migrants' But whenever I try to use q = "migrants%2C%20OR%20migrations%2C%20OR%20refugees%20&geocode=%2244.802416%2C20.465601%2C80mi%22" I get this error print json.dumps(results[0], indent=1) IndexError: list index out of range which I am guessing it means that it is not fining any tweets under these conditions but when I use the same query in Exploring the Twitter API console it is working.

Any ideas what could be wrong please?

— Reply to this email directly or view it on GitHub https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/issues/281 .

[vlaco]

You were right, it works without the url encoding, but geocode was my major problem. Luckily I managed to solve it so I will share the code here in case someone else has the same issue. This code is for a case when you want to search for tweets containing specific words and are posted from a specific country.

import oauth_login
import json

def twitter_search(twitter_api, f, p, max_results=200, **kw):
    result = twitter_api.geo.search(query = p, granularity="country")
    place_id = result['result']['places'][0]['id']
    search_results = twitter_api.search.tweets(q="%s place:%s" % (f, place_id), count=100, **kw)

    statuses = search_results['statuses']

    max_results = min(1000, max_results)
    for _ in range(10): # 10*100 = 1000
        try:
            next_results = search_results['search_metadata']['next_results']
        except KeyError, e: # No more results when next_results doesn't exist
            break

        kwargs = dict([ kv.split('=')
                         for kv in next_results[1:].split("&") ])
        search_results = twitter_api.search.tweets(**kwargs)
        statuses += search_results['statuses']
        if len(statuses) > max_results:
            break

    return statuses

\# Sample usage
twitter_api = oauth_login()
p = "Serbia"

f = 'migrants OR migrations OR refugees'

results = twitter_search(twitter_api, f, p, max_results=10)
\# Show one sample search result by slicing the list...
print json.dumps(results[0], indent=1)

[ptwobrussell]

Awesome, thanks for sending this. I'll plan to add it as a new receipt to the Twitter Cookbook chapter whenever I get around to making updates.

On Fri, Sep 18, 2015 at 12:18 PM, Vanja Vlaco notifications@github.com wrote:

You were right, it works without the url encoding, but geocode was my major problem. Luckily I managed to solve it so I will share the code here in case someone else has the same issue. This code is for a case when you want to search for tweets containing specific words and are posted from a specific country.

import oauth_login import json

def twitter_search(twitter_api, f, p, max_results=200, _kw): result = twitter_api.geo.search(query = p, granularity="country") place_id = result['result']['places'][0]['id'] search_results = twitter_api.search.tweets(q="%s place:%s" % (f, place_id), count=100, _kw)

statuses = search_results['statuses']

max_results = min(1000, max_results)
for _ in range(10): # 10*100 = 1000
    try:
        next_results = search_results['search_metadata']['next_results']
    except KeyError, e: # No more results when next_results doesn't exist
        break

    kwargs = dict([ kv.split('=')
                     for kv in next_results[1:].split("&") ])
    search_results = twitter_api.search.tweets(**kwargs)
    statuses += search_results['statuses']
    if len(statuses) > max_results:
        break

return statuses

Sample usage

twitter_api = oauth_login() p = "USA"

f = 'migrants OR migrations OR refugees'

results = twitter_search(twitter_api, f, p, max_results=10)

Show one sample search result by slicing the list...

print json.dumps(results[0], indent=1)

— Reply to this email directly or view it on GitHub https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/issues/281#issuecomment-141512101 .

[vlaco]

No problem, thank you :)