ldap resolver paged search messageId error with NetIQ eDirectory

fbmei commented 1 week ago

Top-level intent

Import users with the LDAPIdResolver from a NetIQ eDirectory with a limit of 500 users.

Steps to reproduce

1.Set up LDAPIdResolver for NetIQ eDirectory with size limit of 500. 2.Restart eduMFA 3.Open users dashboard and reload users two times in a row.

Expected outcome

500 users should get loaded after the first reload and loaded again after the second.

Actual outcome

The first reload works and 500 users get loaded. After the second reload the list is empty.

Configuration

eduMFA version: 2.2.0
Installation method: PyPI
Python version: 3.9
Operating system: python:3.9 docker image
Webserver: Apache httpd
Token database: sqlite

Log file

[2024-09-26 19:10:33,339][17][140684933904064][DEBUG][edumfa.lib.resolver:191] Exiting get_resolver_object with result <edumfa.lib.resolvers.LDAPIdResolver.IdResolver object at 0x7ff
3c14b21c0>
[2024-09-26 19:10:33,339][17][140684933904064][DEBUG][edumfa.lib.user:723] with this search dictionary: {'username': '*'} 
[2024-09-26 19:10:33,460][17][140684933904064][ERROR][edumfa.lib.user:744] LDAPSocketReceiveError('invalid messageId received')
[2024-09-26 19:10:33,461][17][140684933904064][DEBUG][edumfa.lib.user:745] Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/edumfa/lib/user.py", line 724, in get_user_list
    ulist = y.getUserList(searchDict)
  File "/usr/local/lib/python3.9/site-packages/edumfa/lib/resolvers/LDAPIdResolver.py", line 728, in getUserList
    for entry in ignore_sizelimit_exception(self.l, g):
  File "/usr/local/lib/python3.9/site-packages/edumfa/lib/resolvers/LDAPIdResolver.py", line 180, in ignore_sizelimit_exception
    last_entry = next(generator)
  File "/usr/local/lib/python3.9/site-packages/ldap3/extend/standard/PagedSearch.py", line 56, in paged_search_generator
    result = connection.search(search_base,
  File "/usr/local/lib/python3.9/site-packages/ldap3/core/connection.py", line 853, in search
    response = self.post_send_search(self.send('searchRequest', request, controls))
  File "/usr/local/lib/python3.9/site-packages/ldap3/strategy/restartable.py", line 245, in post_send_search
    raise e
  File "/usr/local/lib/python3.9/site-packages/ldap3/strategy/restartable.py", line 234, in post_send_search
    ret_value = SyncStrategy.post_send_search(self, self.connection.send(self._current_message_type, self._current_request, self._current_controls))
  File "/usr/local/lib/python3.9/site-packages/ldap3/strategy/sync.py", line 178, in post_send_search
    responses, result = self.get_response(message_id)
  File "/usr/local/lib/python3.9/site-packages/ldap3/strategy/base.py", line 355, in get_response
    responses = self._get_response(message_id, timeout)
  File "/usr/local/lib/python3.9/site-packages/ldap3/strategy/sync.py", line 234, in _get_response
    raise LDAPSocketReceiveError(self.connection.last_error)
ldap3.core.exceptions.LDAPSocketReceiveError: invalid messageId received

More Information

I've debugged the issue so far, and I think I found the problem. The LDAP server is not setting the paged search cookie to null when the size limit is reached. This leads to more search requests, which overwhelm the server and cause it to respond with incorrect message IDs. This looks like an issue with NetIQ eDirectory, but it also makes it unusable with eduMFA. I've also observed that, for example, Apache DS enforces its size limit on the client side and stops sending new search requests after the size limit has been hit. I have implemented a fix that handles the paged search cookie on its own and enforces the size limit on the client side to make the implementation more robust against problematic servers. I will attach a pull request when the tests are complete.

fritterhoff commented 1 week ago

Hi,

thats an interesting corner case/bug you found. Sadly I don't have access to an NetIQ directory so I can't reproduce/check. Feel free to propose a PR with a fix!

Thanks in advance.

fbmei commented 4 hours ago

Hi,

I have encountered a problem with my fix. Since it is possible for the LDAP server to send new referrals while performing a paged search, I need to account for that. This results in a near reimplementation of the ldap3.extend.standard.PagedSearch.paged_search_generator method, with only small changes for the size limit check. The code looks so similar to the code from the ldap3 project that I am unsure if I can submit it in that form. This is the first time I am contributing to an open-source project, and I am not sure how to handle such a situation. Since the ldap3 project is licensed under the GPL and eduMFA is under the AGPL, I do not know if it would be possible to use the method from ldap3, modify it to fix the problem, and credit the author from ldap3.

Best regards, Fabian Meier

eduMFA / eduMFA