DirectoryTree / LdapRecord

A fully-featured LDAP framework.
https://ldaprecord.com
MIT License
487 stars 44 forks source link

No way (pagination, slicing) to get more than 500 results [OpenLDAP 2.4] #724

Open OdyX opened 2 months ago

OdyX commented 2 months ago

Environment:

Describe the bug:

In a specific OU on our server, we have more than 500 entries (565 as of today), but there's no way, using the documented methods (or I'm using them wrong), to get all the records, via get() or paginate(). As you can see below, I've found a way to circumvent the limit (by hacking around a loop with a crazy filter), but there's something either broken, or that I don't understand properly.

Could it be a limitation by the server?

<?php

include("./vendor/autoload.php");

use LdapRecord\Container;
use LdapRecord\Connection;

$connectionSettings = [
  "hosts" => ["host.example.com"],
  "port" => 636,
  "base_dn" => "ou=org,dc=example,dc=com",
  "use_ssl" => true,
  "username" => "xxxx",
  "password" => "xxxx",
  "timeout" => 3,
];

$connection = new Connection($connectionSettings);
Container::addConnection($connection);

$clients = $connection->query()
                       ->in('ou=clients,ou=externals,ou=org,dc=liip,dc=ch')
                       ->where('objectClass','=','inetorgperson')
                       ->orderBy('cn');

$clients_ = clone $clients;
// First, let's prove we have more than 500 clients
// The trick is to run the request to the max allowed (get()), get the list of cns and massage it as filter for the next round.
$results = []; 
$filters = [];
do {
    $hadSomeThisTurn = false;
    foreach ($clients_->rawFilter($filters)->get() as $entry) {
        $results[$entry['cn'][0]] = $entry;
        $hadSomeThisTurn = true;
    }
    $filters = ["(!(|(cn=".implode(")(cn=", array_keys($results)).")))"];
} while($hadSomeThisTurn);

// Says 565 (in our case)
dump("fakePaginator", count($results));

// Says "500"
$clients_ = clone $clients;
dump("get()", count($clients_->get()));

assert(!$clients_->isPaginated());

// Says "500" too
$clients_ = clone $clients;
dump("paginate()", count($clients_->paginate()));

// Says "500" too
$clients_ = clone $clients;
dump("paginate(600)", count($clients_->paginate(600)));

// Says "500" too
$clients_ = clone $clients;
dump("paginate(50)", count($clients_->paginate(50)));

assert($clients_->isPaginated());

$clients_ = clone $clients;
$results = [];
$page = 1;
$perPage = 100;
do {
    $slice = $clients_->slice($page, $perPage);
    foreach ($slice->items() as $entry) {
        $results[] = $entry;
    }
    $page++;
} while($slice->hasMorePages()) ;

// Slicing doesn't work, there are 2 pages only
dump("pages", $page);
// Says "500" too
dump("slices", count($results));
stevebauman commented 3 weeks ago

Hi @OdyX,

Are you able to retrieve your OpenLDAP servers olcSizeLimit to see what is currently at?

https://stackoverflow.com/questions/38734417/how-to-set-search-sizelimit-in-openldap-2-4-43-x86

Knowing this may uncover some clues.