DocList only retrieves 100 entries

[GoogleCodeExporter]

What steps will reproduce the problem?

I get only 100 documents in a Feed of Documents.  I've tested this in a 
couple of ways. For example:

1. When getting documents from the constructor of the Feed class;  New 
Feed(of Document)(DocsService, DocListQuery)
2. When using the RetrieveAllUsers method of the DocumentsRequest Class. 
(using both the 2-legged OAuth, and the username/password constructor)

Original issue reported on code.google.com by edwin.la...@gmail.com on 27 Jan 2010 at 6:39

[GoogleCodeExporter]

Just wanted to post my workaround.  Note that I have to replace the 
xoauth_requestor_id query parameter with an empty string to get it to work.  
This 
code is necessary because of behavior described in Issue 314:

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
            Dim docsQuery As DocumentsListQuery = _docsListQuery
            docsQuery.OAuthRequestorId = user
            docsQuery.Owner = user
            docsQuery.ShowFolders = True
            docsQuery.NumberToRetrieve = 100

            Dim docFeed As New Feed(Of Document)(_docsService, docsQuery)
            Dim allEntries As List(Of Document) = docFeed.Entries.ToList

            Do While Not docFeed.AtomFeed.Links.FindService("next", Nothing) Is          
Nothing And docFeed.Entries.Count = 100

                Dim docsQueryNextUri As String = 
docFeed.AtomFeed.Links.FindService("next", Nothing).HRef.ToString
                docsQueryNextUri = docsQueryNextUri.Replace("xoauth_requestor_id=" & 
user, "")
                Dim docsQueryNext As New DocumentsListQuery(docsQueryNextUri)
                docsQueryNext.OAuthRequestorId = user

                docFeed = New Feed(Of Document)(_docsService, docsQueryNext)

                allEntries.AddRange(docFeed.Entries.ToList)

            Loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Original comment by edwin.la...@gmail.com on 28 Jan 2010 at 5:29

[GoogleCodeExporter]

Are you setting the AutoPaging property to true anywhere? The model classes 
(feed, Request etc) are having 
autopaging build in, but it is FALSE by default. So if you do this it should 
get you all entries (beside the issue 
with the oauthrequestor_id, which will probalby make the paging fail).

Frank

Original comment by fman...@gmail.com on 1 Feb 2010 at 9:14

• Changed state: Reviewed

[GoogleCodeExporter]

Hello Frank,

I thought I tried that once, but I didn't give it a full effort.  It could have 
been 
failing due to the oauthrequestor_id issue and I gave up looking for a solution 
to 
the paging problem I was having.

Just to make sure I understand, I can set the Feed.AutoPaging property to true 
after 
I have constructed the Feed Class with the DocListQuery, and it will 
automatically 
fetch new Document entries when I loop through them?

Original comment by edwin.la...@gmail.com on 1 Feb 2010 at 4:47

[GoogleCodeExporter]

That is the basic idea, and it is supposed to work (i have no bugs filed 
against that). There are samples on how to 
do so in the unittests and some other sample apps as well. If it is not working 
right now, i do assume it is either 
due to some service limit (which i doubt) or due to the oauth requestor. Can 
you try with another auth method?

Original comment by fman...@gmail.com on 1 Feb 2010 at 5:01

[GoogleCodeExporter]

I tried the following code:

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

 Dim reqSettings As New RequestSettings("test", "user@example.org", "password")
        Dim docReq As New DocumentsRequest(reqSettings)
        Dim docs As Feed(Of Document) = docReq.GetEverything()
        Console.WriteLine("test1a - " & docs.Entries.Count.ToString)
        Dim count As Integer = 0
        For Each d As Document In docs.Entries
            count += 1
        Next
        Console.WriteLine("test1b - " & count.ToString)

        '''''''''''''''''''
        '''''''''''''''''''
        Dim docsService As New DocumentsService("test")
        docsService.setUserCredentials("user@example.org", "password")
        Dim docsQuery As New DocumentsListQuery()
        docs = New Feed(Of Document)(docsService, docsQuery)
        Console.WriteLine("test2a - " & docs.Entries.Count.ToString)
        count = 0
        For Each d As Document In docs.Entries
            count += 1
        Next
        Console.WriteLine("test2b - " & count.ToString)

        ''''''''''''''''''''
        ''''''''''''''''''''
        Dim docRequestFactory As New GOAuthRequestFactory("cl", "test")
        docRequestFactory.ConsumerKey = "example.org"
        docRequestFactory.ConsumerSecret = "secret"
        docsService = New DocumentsService("test")
        docsService.RequestFactory = docRequestFactory
        docsQuery = New Google.GData.Documents.DocumentsListQuery
        docsQuery.OAuthRequestorId = "user@example.org"
        docs = New Feed(Of Document)(docsService, docsQuery)
        Console.WriteLine("test3a - " & docs.Entries.Count.ToString)
        count = 0
        For Each d As Document In docs.Entries
            count += 1
        Next
        Console.WriteLine("test3b - " & count.ToString)
        Console.ReadLine()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Say I have a total of 170 docs distributed like this:
Folders  - 10
Google Documents(including pdfs) - 150
Other files - 2

If I compile the source code and reference Google.GData.Client and 
Google.GData.Documents, I get the following output:

[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[

test1a - 100
test1b - 100
test2a - 100
test2b - 100
test3a - 100
test3b - 100

]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]

If I reference the Google.GData.Client and Google.GData.Documents dll's in 
client/cs/lib/Release, I get the following:

[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[

test1a - 160
test1b - 160
test2a - 150
test2b - 150
test3a - 150
test3b - 150

]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]

I've always been compiling the code and referencing the result.  The dlls 
aren't 
current with the source.  Is this by design?  If so, this is probably why 
pjdeeb 
submitted Issue 324. 

As always, that's for your quick response!

Original comment by edwin.la...@gmail.com on 3 Feb 2010 at 12:31

[GoogleCodeExporter]

Sorry, In the example above, I meant that there are a total of 162 documents

Original comment by edwin.la...@gmail.com on 3 Feb 2010 at 2:49

[GoogleCodeExporter]

Ok, i just added 120 dummy docs to my account, and run this code (the one in 
trunk, compiled my own dlls, 
using Documents3 project).

            RequestSettings settings = new RequestSettings("DocumentsUpload.NET", "user", "pwd");
            settings.AutoPaging = true;
            DocumentsRequest r = new DocumentsRequest(settings);

            // this returns the server default answer
            Feed<Document> feed = r.GetDocuments();

            int c = 0;
            foreach (Document d in feed.Entries)
            {
                c++;
            }

            MessageBox.Show("Entries counted: " + c);

And this get's me the correct number of entries (126 or so).

The default return by doclist is 100 i think. Because you are never setting 
autopage or following the next link 
when you use the service directly, you never get more than 100.

When you use the old DLLs you probably are using doclist 2.0 instead of 3.0, 
and i think they did not cut off 
at 100 entries.

Original comment by fman...@gmail.com on 3 Feb 2010 at 11:04

• Changed state: Invalid

[GoogleCodeExporter]

Hey Frank,

Thanks for the help here.  I don't get why autopaging is false by default, but 
I'm 
with you now.    I think there still may be an issue here though. 

I called the count property of Feed.Entries, and that makes is so I can't loop 
through the entries:

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
 Dim reqSettings As New RequestSettings("test", "user@example.org", "password")
        reqSettings.AutoPaging = True
        Dim docReq As New DocumentsRequest(reqSettings)
        Dim docs As Feed(Of Document) = docReq.GetEverything()
        Console.WriteLine("test1a - " & docs.Entries.Count.ToString)
        Dim count As Integer = 0
        For Each d As Document In docs.Entries
            count += 1
        Next
        Console.WriteLine("test1b - " & count.ToString)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Produces

[[[[[[[[[[[[[[[[[[[[[
test1a - 120
test1b - 0
]]]]]]]]]]]]]]]]]]]]]

Don't know if this is desired behavior, just though I'd bring it up.

Original comment by edwin.la...@gmail.com on 3 Feb 2010 at 4:46

[GoogleCodeExporter]

 That looks like a bug, as if Entries.Count actually skipped through the end of the feed, and the foreach d as 
document is not starting at the beginning again... 

autopaging is false by default as paging is a costly operation (go to server, 
get next chunk etc).

Frank

Original comment by fman...@gmail.com on 3 Feb 2010 at 5:43

• Changed state: Started

[GoogleCodeExporter]

feed.Entries is an IEnumerable. It does not have a real count. It counts by 
skipping the whole list and counting 
it. So it goes to the server get's all the chunks and ends at the end.

An IEnumerable is a forward only cursor, when we get there the next time in the 
foreach, we are already at the 
end of the feed.

I changed the base code to allow for this. So now you call foreach several 
times, and it will go tback to the 
server each time. 

Note that this is bad performance wise, and inconsistent (you can count your 
docs 3x - and get differfent 
answers if the service changed in that time, like someone deleted a doc).

It would be better to copy the feed entries into an array first.

Fixed in Subversion.

Original comment by fman...@gmail.com on 9 Feb 2010 at 10:16

• Changed state: Fixed