Only return 100 users

[GoogleCodeExporter]

It would be nice to be able to return all your twitter users rather than
the first 100.

There are options in the twitter api
(http://groups.google.com/group/twitter-development-talk/web/api-documentation#U
serMethods)
to return the next 100 friends.  Not sure how this is used though.

I'll try and take a look, but I'm new to python.

Original issue reported on code.google.com by dor...@gmail.com on 11 Jul 2008 at 6:52

[GoogleCodeExporter]

Interesting idea, though beware that I'm very reluctant to accept patches that
automatic trigger multiple queries to the Twitter API.

Original comment by dclinton on 19 Jul 2008 at 2:43

[GoogleCodeExporter]

I suspect most users do not follow more than 200 users, so that would only 
trigger 2
queries.  We could enforce a maximum return say 1000 users, which still would 
only
amount to 10 queries.  Maybe the method could take a parameter which states the
number of users to return.

Original comment by dor...@gmail.com on 19 Jul 2008 at 4:31

[GoogleCodeExporter]

Hi, I'm attaching a simple fix to this issue. The patched functions (listed 
below)
would now carry a new keyword argument 'page' and this will allow one to 
retrieve all
the sets of friends, direct messages, followers, etc.

Here are the functions that have been changed in this patch that I've attached
(twitter_new.py):

* GetFriends(self, user=None)
* GetFollowers(self)
* GetDirectMessage(self, since=None)

With new signatures being:

* GetFriends(self, user=None, page=1)
* GetFollowers(self, user=None, page=1)
* GetDirectMessage(self, since=None, page=1)

As seen, GetFriends now has page support and GetFollowers has both user and 
page support.

To get all Friends, for instance, we could now do the following outside 
twitter.py:

    friends, counter = [], 1
    while not len(friends) % 100:
        friends += api_instance.GetFriends('someuser', page=counter)
        counter += 1

You could then check the length of the friends list to verify if all users have 
been got.

P.s. Also attaching the example retrieving code (sample.py), in case it does not
display well here.

Original comment by qwertymaniac on 22 Dec 2008 at 6:50

Attachments:

• twitter.patch
• sample.py

[GoogleCodeExporter]

Forgot to add, even replies are enhanced.

The following:

* GetReplies(self)

Becomes:

* GetReplies(self, page=1)

Original comment by qwertymaniac on 22 Dec 2008 at 6:52

[GoogleCodeExporter]

Just another vote for the OPTIONAL parameter page=x. I consider myself an 
average
Twitter user and have about two pages of 100 friends and followers.

Original comment by niklassa...@gmail.com on 31 Jan 2009 at 10:08

[GoogleCodeExporter]

PS, thanks for the patch, it helped me out with my problem :-)

Original comment by niklassa...@gmail.com on 31 Jan 2009 at 12:26

[GoogleCodeExporter]

Can anybody tell me how to apply patch in windows? And it seems that 
qwertymaniac
forgot to attach twitter_new.py

Original comment by callsmah...@gmail.com on 13 Feb 2009 at 9:35

[GoogleCodeExporter]

Original comment by dclinton on 21 Feb 2009 at 5:17

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

hi dclinton - are you going to incorporate the "page" option as outlined here? 
It
would be great if you did!

Original comment by dgel...@gmail.com on 28 Feb 2009 at 4:54

[GoogleCodeExporter]

Another option would be to use a yield statement for returning users, which 
would
only trigger only when the caller requested users beyond the initial 100.

Assuming we already know the number of followers (which Twitter gives us), we 
can do
something like this:

page = 0
while (page * 100) < follower_count:
    url = 'http://twitter.com/statuses/followers.json?page=%d' % (page + 1)
    json = self._FetchUrl(url)
    data = simplejson.loads(json)
    self._CheckForTwitterError(data)
    page += 1
    for x in data:
        yield User.NewFromJsonDict(x)

(Note! this is untested code, but I've used similar functions in home-brew 
scripts
and they seem to work just fine.)

Similar methods could also be written for any Twitter API method that could
potentially use the page parameter.

Original comment by chris.mu...@gmail.com on 7 Mar 2009 at 9:01

[GoogleCodeExporter]

Something like this seems necessary.  One way or another, the python API needs 
to be able to get the full 
complement of users (or other information which is "doled out" by the Twitter 
API)

Original comment by joegermu...@gmail.com on 9 Mar 2009 at 6:21

[GoogleCodeExporter]

I agree that this is needed. I just started using python-twitter and this was 
the
very first thing I bumped into. I have 500+ followers ... I guess I will try 
the patch. 

Original comment by robert.l...@gmail.com on 11 Mar 2009 at 1:31

[GoogleCodeExporter]

RE: Comment 3. Patch is against older version of twitter-python. I just tried 
it on
Rev 136 with following results. 
boblq@boblq-desktop:~/Projects/python-twitter/python-twitter$ patch -p5 
<twitter.patch 
patching file twitter.py
Hunk #1 FAILED at 1053.
Hunk #2 FAILED at 1061.
Hunk #3 FAILED at 1083.
Hunk #4 succeeded at 1580 with fuzz 1 (offset 480 lines).
Hunk #5 FAILED at 1617.
Hunk #6 FAILED at 1636.
5 out of 6 hunks FAILED -- saving rejects to file twitter.py.rej
boblq@boblq-desktop:~/Projects/python-twitter/python-twitter$ ls -l twitter.py*
-rwxr-xr-x 1 boblq boblq 67502 2009-03-11 12:02 twitter.py
-rwxr-xr-x 1 boblq boblq 67375 2009-03-11 12:01 twitter.py.orig
-rw-r--r-- 1 boblq boblq  4035 2009-03-11 12:02 twitter.py.rej
boblq@boblq-desktop:~/Projects/python-twitter/python-twitter$ 

Do we want to integrate this into the main trunk? 

Original comment by robert.l...@gmail.com on 11 Mar 2009 at 8:07

[GoogleCodeExporter]

Absolutely, this is a necessary addition.  If you don't have over 100
followers/followees you're not even a serious Twitterer.  Well, let me explain 
...

There are two kinds of people who use Twitter: People who only follow close 
friends
whose every tweet they intend on reading, and social networking people who 
"build
their web" by following LOTS of people.  The former type won't likely run into 
the
100 user limit, while the latter type will run into it within a couple days of
account creation.

Personally, I have two Twitter accounts, one that just follows close friends 
and one
that does the whole social networking thing (which is also the account I wrote 
the
auto-follow script for), and Python-Twitter is of limited utility for the second
account at the moment.  We already have the patch here to fix it, so let's do 
it.

Original comment by cydeweys on 11 Mar 2009 at 8:28

[GoogleCodeExporter]

OK. I made a pass at updating the patch originally submitted by querymaniac.
(Comments 3 & 4)

This patch breaks four of the unit tests. I looked at twitter_test.py, tried to 
fix
the tests, but did not understand the code. My test mojo is broken :( 

I did some minimal testing against my own twitter account and it seems to work
correctly. 

Maybe this will be of some use, but it should be tested more thoroughly and the 
unit
tests need to be fixed. (Also docs :) 

Still it is a start ... publish early, publish often. 

BobLQ

Original comment by robert.l...@gmail.com on 11 Mar 2009 at 10:08

Attachments:

• twitter.patch

[GoogleCodeExporter]

Re Comment 15: The patch is against svn 136. 

Original comment by robert.l...@gmail.com on 11 Mar 2009 at 10:12

[GoogleCodeExporter]

Okay, I'm looking at the patch now and I see those four failed tests.  I should 
be
able to fix the tests pretty simply and have a patch for the tests as well.

The only problem is this patch has some (very minor, trivial to resolve) 
conflicts
with my recently submitted patch for support for non-Twitter sites.  The longer 
we
leave all these patches hanging around without committing them, the uglier the
divergences get.  That's why quick committal is important, and why I'm angling 
for
commit access.

Original comment by cydeweys on 12 Mar 2009 at 12:02

[GoogleCodeExporter]

Just to be sure I understand, this change is simply to enable the 'page' 
parameter,
not to automatically make requests for multiple pages, right?

Original comment by dclinton on 12 Mar 2009 at 12:09

[GoogleCodeExporter]

Okay, here's a fix for the tests.  The issue patch #20 now passes all tests.  I 
had
to append the new page parameter to the end of each URL that the testing 
program was
expecting.  I assert that this is correct because this is the same way that the
testGetPublicTimeline works.

Original comment by cydeweys on 12 Mar 2009 at 12:16

Attachments:

• twitter_test_pagination.patch

[GoogleCodeExporter]

And yes dclinton, this patch simply allows the page parameter; it does not
automatically request multiple pages of results.

Doing so will be an option for a future patch (not a default option, mind you, 
but
it's probably a good idea to handle summation of paged results once in the 
library
instead of repeatedly across a whole number of different programs using the 
library).

Original comment by cydeweys on 12 Mar 2009 at 12:18

[GoogleCodeExporter]

Whoops, I accidentally included the page patch for twitter.py in my diff as 
well as
the fixes to the test.  Ah well, I guess that just makes it more convenient to 
apply
and commit, eh?  :-D

Original comment by cydeweys on 12 Mar 2009 at 12:20

[GoogleCodeExporter]

Question about test_pagination.patch 

Why does the 
 def testGetReplies(self): 
not change to 
 def testGetReplies(self, page):

Or some such?

Do you have a url to a good tutorial on the python unit test framework (Else I 
Google)

Original comment by robert.l...@gmail.com on 12 Mar 2009 at 12:26

[GoogleCodeExporter]

Robert, I don't have a good URL, but let me try to explain.

Each function that starts with test... and then has a name of the function from 
the
PyTwitter library is a complete, self-contained, separate test that is called
directly by the unit testing framework.  As such, it can have no parameters, as 
the
test framework wouldn't know what to do with them.  We ensure that the page 
parameter
is handled properly within the unit test method.

Original comment by cydeweys on 12 Mar 2009 at 12:54

[GoogleCodeExporter]

I checked in a variant of the 'page' parameter patch in r137.  Please verify:

  http://code.google.com/p/python-twitter/source/detail?r=137

Will apply the test patch and mark this resolved.

Original comment by dclinton on 12 Mar 2009 at 1:05

[GoogleCodeExporter]

A modified test patch applied in r138:

  http://code.google.com/p/python-twitter/source/detail?r=138

Marking fixed.  Please verify.

And thanks for the patches!

Original comment by dclinton on 12 Mar 2009 at 1:12

• Changed state: Fixed

[GoogleCodeExporter]

Looks correct to me.  Thanks dclinton.

Original comment by cydeweys on 12 Mar 2009 at 1:25

[GoogleCodeExporter]

RE Comment 23. Ok, I got it. Pretty obvious when you look at the changed code
http://code.google.com/p/python-twitter/source/detail?r=138 Thanks Cyde.

Original comment by robert.l...@gmail.com on 12 Mar 2009 at 3:14

[GoogleCodeExporter]

Nice to get this one fixed.

Thanks!

Original comment by dor...@gmail.com on 12 Mar 2009 at 8:27