Matomo ldap sync users upper limit is 999

[lizzyliao]

Hi All: matomo ldap sync users upper limit is 999, I can not sync any ldap users into matomo,

How do I solve this problem ?
Thank you very much.

• Matomo Version: 4.7.1
• PHP Version: PHP 7.4.30
• Server Operating System: centos7
• Additionally installed plugins: ldap

[AltamashShaikh]

@lizzyliao what error are you getting ? And are you using the ./console loginldap:synchronize-users command to sync users ?

[lizzyliao]

Hi AltamashShaikh : Yes, use ./console loginldap:synchronize-users command.

[AltamashShaikh]

@lizzyliao What error do you get ?

[lizzyliao]

@AltamashShaikh No error message shown. Sync successfully. But just only "Synchronized 999 users!" Thank you very much~

[AltamashShaikh]

@lizzyliao I do not see any hard limits in code, may be you can try running the command using ./console loginldap:synchronize-users -vvv and see if there is any error message or any other debug message helpful for us to debug this issue.

[lizzyliao]

@AltamashShaikh We see error messasge as following: "Synchronized 999 users! Could not synchronize the following users in LDAP: K21050416 Could not instantiate mail function."

But we can see 'K21050416" already in matomo web ui. Administration -> System -> users The users page show 'K21050416' with correct email "xxx@xxx.com"

The program stopped and some other users can not sync into matomo.

Thank you very much

[lizzyliao]

@AltamashShaikh

we use debug flag log as following: DEBUG [2022-10-06 02:28:31] 26238 ldap_search result is [resource] DEBUG [2022-10-06 02:28:31] 26238 Calling ldap_get_entries([resource], [resource]) DEBUG [2022-10-06 02:28:31] 26238 ldap_get_entries result is not null DEBUG [2022-10-06 02:28:31] 26238 Calling ldap_close([resource]) DEBUG [2022-10-06 02:28:31] 26238 ldap_close returned true DEBUG [2022-10-06 02:28:31] 26238 Model\LdapUsers: end getUser() with array["objectclass","cn","sn","descriptio n","distinguishedname","instancetype","whencreated","whenchanged","displayname","usncreated","info","memberof"," usnchanged","proxyaddresses","homemdb","submissioncontlength","garbagecollperiod","mdbusedefaults","mailnickname ","protocolsettings","internetencoding","name","objectguid","useraccountcontrol","pwdlastset","primarygroupid"," objectsid","samaccountname","samaccounttype","showinaddressbook","legacyexchangedn","userprincipalname","objectc ategory","dscorepropagationdata","lastlogontimestamp","mail","thumbnailphoto","msexchpoliciesexcluded","msexchom aadminwirelessenable","msexchhomeservername","msexchmailboxsecuritydescriptor","msexchuseraccountcontrol","msexc hmailboxguid","msexchmailboxfolderset","msexchtransportrecipientsettingsflags","msexchumdtmfmap","msexchmdbrules quota","msexchaddressbookflags","msexchprovisioningflags","msexchmailboxtemplatelink","msexchumenabledflags2","m sexchwhenmailboxcreated","msexchrecipientdisplaytype","msexchmailboxauditenable","msexchrbacpolicylink","msexchr ecipientsoftdeletedstatus","msexchcalendarloggingquota","msexchversion","msexchmailboxauditlogagelimit","msexchr ecipienttypedetails","msexchdumpsterquota","msds-externaldirectoryobjectid","msexchdumpsterwarningquota","msexch moderationflags","msexcharchivequota","msexcharchivewarnquota","msexchelcmailboxflags","msexchbypassaudit","msex chtextmessagingstate","msexchgroupsecurityflags","dn"] DEBUG [2022-10-06 02:28:31] 26238 UserSynchronizer::synchronizeLdapUser: synchronizing user [ piwik login = Wit s.KevinHuang, ldap login = Wits.KevinHuang ] success!

Synchronized 1000 users!

Error: error or warning logs detected, exit 1

[AltamashShaikh]

@AltamashShaikh We see error messasge as following: "Synchronized 999 users! Could not synchronize the following users in LDAP: K21050416 Could not instantiate mail function."

But we can see 'K21050416" already in matomo web ui. Administration -> System -> users The users page show 'K21050416' with correct email "xxx@wistron.com"

The program stopped and some other users can not sync into matomo.

Thank you very much

The error "Could not instantiate mail function" is a error from PHPMailer, can you check if you have setup the SMTP server correctly ?

[lizzyliao]

@AltamashShaikh We have never set up an smtp server. Will this affect sync ldap users?

I type "./console loginldap:synchronize-users" command again. The error message "K21050416 Could not instantiate mail function" do not show again. The error message is as following: _

"EBUG [2022-10-06 06:19:52] 126938 ldap_bind result is '1' DEBUG [2022-10-06 06:19:52] 126938 Calling ldap_search([resource], 'dc=xxxx', '(&(&(objectClass=Person)(memb erOf:1.2.840.113556.1.4.1941:=cn=matomopaas,ou=Group_Object,dc=wih,dc=xxxx) )(samaccountname=Wits.KevinHuang) )') DEBUG [2022-10-06 06:19:52] 126938 ldap_search result is [resource] DEBUG [2022-10-06 06:19:52] 126938 Calling ldap_get_entries([resource], [resource]) DEBUG [2022-10-06 06:19:52] 126938 ldap_get_entries result is not null DEBUG [2022-10-06 06:19:52] 126938 Calling ldap_close([resource]) DEBUG [2022-10-06 06:19:52] 126938 ldap_close returned true DEBUG [2022-10-06 06:19:52] 126938 Model\LdapUsers: end getUser() with array["objectclass","cn","sn","descripti on","distinguishedname","instancetype","whencreated","whenchanged","displayname","usncreated","info","memberof", "usnchanged","proxyaddresses","homemdb","submissioncontlength","garbagecollperiod","mdbusedefaults","mailnicknam e","protocolsettings","internetencoding","name","objectguid","useraccountcontrol","pwdlastset","primarygroupid", "objectsid","samaccountname","samaccounttype","showinaddressbook","legacyexchangedn","userprincipalname","object category","dscorepropagationdata","lastlogontimestamp","mail","thumbnailphoto","msexchpoliciesexcluded","msexcho maadminwirelessenable","msexchhomeservername","msexchmailboxsecuritydescriptor","msexchuseraccountcontrol","msex chmailboxguid","msexchmailboxfolderset","msexchtransportrecipientsettingsflags","msexchumdtmfmap","msexchmdbrule squota","msexchaddressbookflags","msexchprovisioningflags","msexchmailboxtemplatelink","msexchumenabledflags2"," msexchwhenmailboxcreated","msexchrecipientdisplaytype","msexchmailboxauditenable","msexchrbacpolicylink","msexch recipientsoftdeletedstatus","msexchcalendarloggingquota","msexchversion","msexchmailboxauditlogagelimit","msexch recipienttypedetails","msexchdumpsterquota","msds-externaldirectoryobjectid","msexchdumpsterwarningquota","msexc hmoderationflags","msexcharchivequota","msexcharchivewarnquota","msexchelcmailboxflags","msexchbypassaudit","mse xchtextmessagingstate","msexchgroupsecurityflags","dn"] DEBUG [2022-10-06 06:19:52] 126938 UserSynchronizer::synchronizeLdapUser: synchronizing user [ piwik login = Wi ts.KevinHuang, ldap login = Wits.KevinHuang ] success!

Synchronized 1000 users!

Error: error or warning logs detected, exit 1"

/var/log/cron

Oct 6 14:01:01 matomo-dev run-parts(/etc/cron.hourly)[126575]: starting 0anacron Oct 6 14:01:01 matomo-dev run-parts(/etc/cron.hourly)[126584]: finished 0anacron Oct 6 14:01:01 matomo-dev run-parts(/etc/cron.hourly)[126575]: starting mcelog.cron Oct 6 14:01:01 matomo-dev run-parts(/etc/cron.hourly)[126590]: finished mcelog.cron Oct 6 14:10:01 matomo-dev CROND[126749]: (root) CMD (/usr/lib64/sa/sa1 1 1) Oct 6 14:10:01 matomo-dev CROND[126751]: (root) CMD (/matomo/matomo/console loginldap:synchronize-users) Oct 6 14:10:26 matomo-dev CROND[126748]: (root) MAIL (mailed 153215 bytes of output but got status 0x004b#012) Oct 6 14:20:01 matomo-dev CROND[126941]: (root) CMD (/matomo/matomo/console loginldap:synchronize-users) Oct 6 14:20:01 matomo-dev CROND[126942]: (root) CMD (/usr/lib64/sa/sa1 1 1) Oct 6 14:20:34 matomo-dev CROND[126940]: (root) MAIL (mailed 153215 bytes of output but got status 0x004b#012)

Thank you very much~

[AltamashShaikh]

@lizzyliao Thanks for the log, I will check more on this and for now no need to update/change anything. I will get back to you on this.

[AltamashShaikh]

@lizzyliao How many users do you have for sync ?

[lizzyliao]

@AltamashShaikh I think maybe about 1100 people, but the number will increase over time.

Thank you very much.

[AltamashShaikh]

@lizzyliao When you re run it does it sync all the 1100 or is it stopping at 1000 only ?

[lizzyliao]

@AltamashShaikh The matomo server is stopping at 1000 only with error messages "Error: error or warning logs detected, exit 1"

Thank you very much.

[AltamashShaikh]

@lizzyliao Ill try to remove my SMTP settings and try to sync a new user and will see if it creates any issue

[sgiehl]

@lizzyliao @AltamashShaikh might that be an issue of the ldap server? I actually haven't done much with ldap the last years, but if I remember correctly there was some sort of "security" policy to limit the number of results. Was is called maxpagesize or so? 🤔

[AltamashShaikh]

@sgiehl Thanks I was not aware of this limit and thought it was exiting due to mailer settings. Its indeed a security feature by LDAP to prevent DDOS. @lizzyliao @sgiehl is correct there is a limit Refer this blog article which explains why its being limited to 1000

[AltamashShaikh]

@lizzyliao Is there anything you need help with it ? Or should we close this issue ?

[lizzyliao]

@sgiehl @AltamashShaikh
Thank you for your information, I will contact our ldap server admin. Thank you very much.

[lizzyliao]

@AltamashShaikh Hi AltamashShaikh (1) Our ldap admin say ldap query default = 1000, but you can query more than 1000 users when you query ldap. (2) I modify plugins/LoginLdap/Ldap/Client.php in matomo system, I add three lines as following: It works. It only can sync 100 users. But set pageSize =2000, it only sync 1000 users.

  $pageSize = 100;
  $cookie = '';        
  ldap_control_paged_result($connectionResource, $pageSize, true, $cookie);
  $result = ldap_search($connectionResource, $baseDn, $ldapFilter, $attributes);

(3) I think If we need sync more than 1000 users, we maybe need use while loop , please see the following url content.
https://stackoverflow.com/questions/8636375/php-ldap-search-size-limit-exceeded

 Thank you very much

[AltamashShaikh]

Hey @lizzyliao Thanks for checking and posting above solution, but ldap_control_paged_result is deprecated in PHP 7.4 and removed in PHP 8.0. I will check what other alternatives we have, can you update the LDAP server setting at your end to fetch more than 1000 records ? We can latter set the sizeLimit as 0 to fetch all the records - Refer https://www.php.net/manual/en/function.ldap-search.php

We need to replace this line https://github.com/matomo-org/plugin-LoginLdap/blob/4.x-dev/Ldap/Client.php#L382 with below line

$result = ldap_search($connectionResource, $baseDn, $ldapFilter, $attributes, $attributes_only = 0, $sizelimit = 0);

[AltamashShaikh]

@lizzyliao If you are gonna try above fix, you need to

1. You need to update your LDAP server to return more than 1000 records
2. Replace the above line as suggested and try and it should work.

[lizzyliao]

@AltamashShaikh I write this program for testing. I can sync 1349 users.

$i=0;
 do {
                $result  = ldap_search($conn, $dn, $filter, $justthese,0,-1,0,LDAP_DEREF_NEVER,[['oid' => LDAP_CONTROL_PAGEDRESULTS, 'value' => ['size' => 1000, 'cookie' => $cookie]]]);
                ldap_parse_result($conn, $result, $errcode , $matcheddn , $errmsg , $referrals, $controls);
                $entries = ldap_get_entries($conn, $result);
            foreach ($entries as $entry) {
                echo "cn: ".$entry['cn'][0]."\n";
                $i++;
            }
            if (isset($controls[LDAP_CONTROL_PAGEDRESULTS]['value']['cookie'])) {
                // Vous devez passer le cookie du dernier appel au prochain
                $cookie = $controls[LDAP_CONTROL_PAGEDRESULTS]['value']['cookie'];
            } else {
                $cookie = '';
            }

 } while (!empty($cookie));

 echo "i= ".$i

I think need use this parameter "LDAP_CONTROL_PAGEDRESULTS"

[lizzyliao]

@AltamashShaikh I modify Client.php, it still only can sync 1000 users. $result = ldap_search($connectionResource, $baseDn, $ldapFilter, $attributes, $attributes_only = 0, $sizelimit = 0); I use parameter "LDAP_CONTROL_PAGEDRESULTS" and cookie , It can sync more than 1000 users in my testing program.
But I modify Client.php according to my testing program, It can not work. I need your help~~ Thank you very much.

[AltamashShaikh]

Okay, I would check the above code you shared if we can add that and won't create any regressions, but this will take time as I need to prioritize it

[lizzyliao]

@AltamashShaikh

OK, Thank you very much. :)

[AltamashShaikh]

@lizzyliao just 1 Q did you update the limit from 1000 to your desired number in your LDAP server?

[lizzyliao]

@AltamashShaikh NO, We don't do any change in our LDAP server.

[AltamashShaikh]

@lizzyliao I have added this task for prioritisation, so that our product team can put it into existing workload

[lizzyliao]

@AltamashShaikh Thank you very much.

[heurteph-ei]

Hi @mattab, @sgiehl, @AltamashShaikh Is it possible to remove the label "answered" if this ticket, as there is still things to do to complete it? I think also that increase the number of LDAP responses is not a good solution, as this can downgrade the LDAP security. Do you know when you'll be able to work on this ticket?

[AltamashShaikh]

@heurteph-ei We currently haven't scheduled this one as we have other priority items scheduled, will update here once we have any update.

[heurteph-ei]

Hi @mattab, @sgiehl, @AltamashShaikh Is it possible to remove the label "answered" if this ticket, as there is still things to do to complete it?

[AltamashShaikh]

@heurteph-ei updated :+1: