currently I am trying to import a huge mount of users (> 7000) using the import and update planer task of the LDAP extension. My problem is, that not all users are imported or updated correctly.
By taking a look inside the code of the LdapImporter.php class and the responsible getUsers(...) method, I figured out, that your extension tries to get all users by one query. If the LDAP query limit exceeds (like in my case), the extension tries to query in a loop all users which ends with the characters *a, *b, ..., *z etc. to minimize the results. This works as long as the partial queries again return less than 1000 users. In my case I have partial results where the LDAP query limit still exceeds (> 1000 entries). If this happened no users are imported or updated for the current character, e.g. *e.
The following code shows a scalable solution for this problem, performing a recursive call if the LDAP query limit exceeds. E.g. when the result limit for the character *e exceeds. A recursive method call is performed to query all users with *ae, *be, ..., *ze etc. Please note that my colleague and I added an optional third parameter to this method to perform the recursive call (no API change).
class LdapImporter {
/**
* retrieves user records from LDAP
*
* @param string $runIdentifier
* @param string $command
* @param string $search
*/
private function getUsers($runIdentifier, $command, $search = '*') {
$ldapUsers = $this->ldapServer->getUsers( $search, false );
if (is_array($ldapUsers)) {
switch ($command) {
case 'import':
$this->storeNewUsers($runIdentifier, $ldapUsers);
break;
case 'update':
$this->updateUsers($runIdentifier, $ldapUsers);
break;
case 'importOrUpdate':
$this->storeUsers($runIdentifier, $ldapUsers);
break;
}
} else {
// recursive search
if ($this->ldapConfig->logLevel) {
$msg = 'LDAP query limit exceeded';
\TYPO3\CMS\Core\Utility\GeneralUtility::devLog($msg, 'ldap', 0);
}
$searchCharacters = \NormanSeibert\Ldap\Utility\Helpers::getSearchCharacterRange();
foreach ($searchCharacters as $thisCharacter) {
$newSearch = substr_replace($search, $thisCharacter, 1, 0);
$msg = 'Query server: ' . $this->ldapServer->getConfiguration()->getUid() . ' with getUsers("' . $newSearch . '")';
if ($this->ldapConfig->logLevel > 1) {
\TYPO3\CMS\Core\Utility\GeneralUtility::devLog($msg, 'ldap', 0);
}
$this->getUsers($runIdentifier, $command, $newSearch);
}
}
}
}
P.S. I am working with the bugfix_13 branch, based on version 3.1.29.
Please let me know if you need further information.
Hi Norman,
currently I am trying to import a huge mount of users (> 7000) using the import and update planer task of the LDAP extension. My problem is, that not all users are imported or updated correctly.
By taking a look inside the code of the
LdapImporter.phpclass and the responsiblegetUsers(...)method, I figured out, that your extension tries to get all users by one query. If the LDAP query limit exceeds (like in my case), the extension tries to query in a loop all users which ends with the characters*a,*b, ...,*zetc. to minimize the results. This works as long as the partial queries again return less than 1000 users. In my case I have partial results where the LDAP query limit still exceeds (> 1000 entries). If this happened no users are imported or updated for the current character, e.g.*e.The following code shows a scalable solution for this problem, performing a recursive call if the LDAP query limit exceeds. E.g. when the result limit for the character
*eexceeds. A recursive method call is performed to query all users with*ae,*be, ...,*zeetc. Please note that my colleague and I added an optional third parameter to this method to perform the recursive call (no API change).P.S. I am working with the bugfix_13 branch, based on version 3.1.29.
Please let me know if you need further information.
Best greetings, Nico