Paging not working as expected

[ssteiner]

Sorry for the double post.. but since codeplex is being shut down..

I seem to have run into a snag with paging. I know Active Directory usually has a maximum page size of 1000. So, in my code, I'm using query.InPagesOf(900) just to be on the safe side.

When I first wrote that, I tested in my lab that has about 100 users. So I set query.InPagesOf(10), and verified that I'd get all results (I do). So I figured things work as desired.

Now I'm in the productive environment, and query.InPagesOf(900) gets me 900 results. query.InPagesOf(100) gets me 100 results, query.InPagesOf(1000) gets me 1000 results, and so forth. So basically, the value I give to InPagesOf is the maximum number of results I get.

When I change the code to automatic paging, so query.ToList(), then I get 500 results (I guess that's the default page size then). But that leaves out everything above 500.

Then I tried manual paging as described here: https://linqtoldap.codeplex.com/wikipage?title=Paging

But, page.HasNextPage is always false.

So, how do I get all results without blowing up whatever maximum search result size the LDAP server has set?

As a workaround, I wrote my own pagingg

private List<T> readPagedManual<T>(IQueryable<T> query, int pageSize)
    {
        if (pageSize > 0)
        {
            List<T> results = new List<T>();
            int nbPages = 1;
            List<T> page = query.Take(pageSize).ToList();
            results.AddRange(page);
            while (page.Count == pageSize)
            {
                page = query.Skip(nbPages * pageSize).Take(pageSize).ToList();
                results.AddRange(page.);
                nbPages++;
            }
            return results;
        }
        else
        {
            return query.ToList();
        }
    }

But that requires server side ordering which may be disabled so that's not really a solution either.

[madhatter22]

Sorry for the very late reply. You are most likely getting different results because the paging control needs to be enabled. I do not know the name of the control off the top of my head.

[ssteiner]

Well, I'm happy you're still around.

Speaking of controls, isn't the paging control enabled by default? After all, I have this OpenLDAP server to work against, where I can only run queries against if I call DisablePaging() on my configuration. And after all, for small data sets, it works perfectly against active directory.

And, I can't seem to talk my LDAP into going without encryption to actually look what controls it offers.

Any chance you could have another look?

[madhatter22]

Still around, but very busy with my day job. Glad you're still making use of the library. Yes, I'll take a second look. I'll try to stand up an instance of OpenLDAP to reproduce the issue.

[ssteiner]

Is there anything I can do to help from my side? I have two Active Directories at my disposal, one being our corporate directory that has plenty of entries.

Just give me a few pointers where I need to look at and I'll step through the code - I already have the source on my box anyway and I'm even currently making builds directly from the source rather than the Nuget.

[ssteiner]

So, I think I stumbled upon something important: For my large queries, I want to get all object of a certain type from my Active directory. So I found this article https://ilminator.wordpress.com/2012/01/27/retrieving-move-than-500-objects-with-linqtoldap/

.WithControls(new[] { new SearchOptionsControl(SearchOption.DomainScope) })

Is where the magic lies. This also increased the size of my search results (without it it was cut to 10'000 user objects in my prod domain).

the only question remaining is whether it is actually possible to really skip a number of results in paging... looking at what happens in Wireshark, if I page, and I want to read page 3, I see all three pages being loaded with the results from the third being returned (using the Skip/Take approach:

Result = query.InPagesOf(pageSize).Skip((page - 1) * pageSize).Take(pageSize).ToList();

) My guess is the answer is no.

[madhatter22]

Your suspicion is correct. Default paging is a black box for LDAP. You have to use a cookie returned by the server to load remaining pages and that cookie may expire if you wait too long when loading the next page. Skip / Take does work when using Virtual List Views, but you have to explicitly enable the feature in the directory.