search

noirello / bonsai

Simple Python 3 module for LDAP, using libldap2 and winldap C libraries.
MIT License
117 stars 33 forks source link

paged_search, not working correctly #37

Closed schlitzered closed 3 years ago

schlitzered commented 4 years ago

hey,

i have two different AD REALMS, one of them will only return exactly 1000 results for paged searches. where the other will return me way more results.

i already talked to the AD maintainers, and they cannot find an issue.

also ldapsearch will return the right result.

i also tried the "ldap3" python client. this client also returns the correct number of results.

any clue what is going on here?

here is some example code:

ldap3:

from ldap3 import Server, Connection
import ldap3

server = Server('dcexample.com', use_ssl=False, get_info=ldap3.ALL)
conn = Connection(server, user='user@example.com', password='password', auto_bind=True)

entries = conn.extend.standard.paged_search('dc=example,dc=com', '(objectClass=group)', attributes=['cn'], paged_size=1000)

results = list()

for entry in entries:
    results.append(entries)

print(len(results))

bonsai

import bonsai

client = bonsai.LDAPClient('ldap://dc.example.com')
client.set_credentials('SIMPLE', user='user@example.com', password='password')
conn = client.connect()
groups = conn.paged_search('DC=example,DC=com', bonsai.LDAPSearchScope.SUBTREE, "(objectClass=user)", page_size=1000)

results = []

for i in groups:
    results.append(i)
print(len(results))

i guess there is a configuration difference between the two AD realms, but since "ldapsearch" and the "ldap3" client return the right number of results, i feel like bonsai is doing something wrong here.

noirello commented 4 years ago

Hi, I'll look into this and try my best to figure out what's happening. What version and which platform are you using?

schlitzered commented 4 years ago

python 3.6 on centos 7.7 and macos with python3.7 had this issue.

i can also provide you the ldap server info of these two AD controller if you like.

noirello commented 4 years ago

Having the server info can't hurt. I'm interested in the module version as well.

schlitzered commented 4 years ago

bonsai version is 1.2.0

noirello commented 4 years ago

Thank you for these.

schlitzered commented 4 years ago

here is the server info for both ldap servers:

first the one where pagination works with bonsai, followed by the server info, where pagination will not work with bonsai, but with other clients:

pagination working on this AD REALM with bonsai:

DSA info (from DSE):
  Supported LDAP versions: 3, 2
  Naming contexts: 
    DC=working,DC=example,DC=com
    CN=Configuration,DC=working,DC=example,DC=com
    CN=Schema,CN=Configuration,DC=working,DC=example,DC=com
    DC=DomainDnsZones,DC=working,DC=example,DC=com
    DC=ForestDnsZones,DC=working,DC=example,DC=com
  Supported controls: 
    1.2.840.113556.1.4.1338 - Verify name - Control - MICROSOFT
    1.2.840.113556.1.4.1339 - Domain scope - Control - MICROSOFT
    1.2.840.113556.1.4.1340 - Search options - Control - MICROSOFT
    1.2.840.113556.1.4.1341 - RODC DCPROMO - Control - MICROSOFT
    1.2.840.113556.1.4.1413 - Permissive modify - Control - MICROSOFT
    1.2.840.113556.1.4.1504 - Attribute scoped query - Control - MICROSOFT
    1.2.840.113556.1.4.1852 - User quota - Control - MICROSOFT
    1.2.840.113556.1.4.1907 - Server shutdown notify - Control - MICROSOFT
    1.2.840.113556.1.4.1948 - Range retrieval no error - Control - MICROSOFT
    1.2.840.113556.1.4.1974 - Server force update - Control - MICROSOFT
    1.2.840.113556.1.4.2026 - Input DN - Control - MICROSOFT
    1.2.840.113556.1.4.2064 - Show recycled - Control - MICROSOFT
    1.2.840.113556.1.4.2065 - Show deactivated link - Control - MICROSOFT
    1.2.840.113556.1.4.2066 - Policy hints [DEPRECATED] - Control - MICROSOFT
    1.2.840.113556.1.4.2090 - DirSync EX - Control - MICROSOFT
    1.2.840.113556.1.4.2204 - Tree deleted EX - Control - MICROSOFT
    1.2.840.113556.1.4.2205 - Updates stats - Control - MICROSOFT
    1.2.840.113556.1.4.2206 - Search hints - Control - MICROSOFT
    1.2.840.113556.1.4.2211 - Expected entry count - Control - MICROSOFT
    1.2.840.113556.1.4.2239 - Policy hints - Control - MICROSOFT
    1.2.840.113556.1.4.2255 - Set owner - Control - MICROSOFT
    1.2.840.113556.1.4.2256 - Bypass quota - Control - MICROSOFT
    1.2.840.113556.1.4.319 - LDAP Simple Paged Results - Control - RFC2696
    1.2.840.113556.1.4.417 - LDAP server show deleted objects - Control - MICROSOFT
    1.2.840.113556.1.4.473 - Sort Request - Control - RFC2891
    1.2.840.113556.1.4.474 - Sort Response - Control - RFC2891
    1.2.840.113556.1.4.521 - Cross-domain move - Control - MICROSOFT
    1.2.840.113556.1.4.528 - Server search notification - Control - MICROSOFT
    1.2.840.113556.1.4.529 - Extended DN - Control - MICROSOFT
    1.2.840.113556.1.4.619 - Lazy commit - Control - MICROSOFT
    1.2.840.113556.1.4.801 - Security descriptor flags - Control - MICROSOFT
    1.2.840.113556.1.4.802 - Range option - Control - MICROSOFT
    1.2.840.113556.1.4.805 - Tree delete - Control - MICROSOFT
    1.2.840.113556.1.4.841 - Directory synchronization - Control - MICROSOFT
    1.2.840.113556.1.4.970 - Get stats - Control - MICROSOFT
    2.16.840.1.113730.3.4.10 - Virtual List View Response - Control - IETF
    2.16.840.1.113730.3.4.9 - Virtual List View Request - Control - IETF
  Supported extensions: 
    1.2.840.113556.1.4.1781 - Fast concurrent bind - Extension - MICROSOFT
    1.2.840.113556.1.4.2212 - Batch request - Extension - MICROSOFT
    1.3.6.1.4.1.1466.101.119.1 - Dynamic Refresh - Extension - RFC2589
    1.3.6.1.4.1.1466.20037 - StartTLS - Extension - RFC4511-RFC4513
    1.3.6.1.4.1.4203.1.11.3 - Who am I - Extension - RFC4532
  Supported features: 
    1.2.840.113556.1.4.1670 - Active directory V51 - Feature - MICROSOFT
    1.2.840.113556.1.4.1791 - Active directory LDAP Integration - Feature - MICROSOFT
    1.2.840.113556.1.4.1935 - Active directory V60 - Feature - MICROSOFT
    1.2.840.113556.1.4.2080 - Active directory V61 R2 - Feature - MICROSOFT
    1.2.840.113556.1.4.2237 - Active directory W8 - Feature - MICROSOFT
    1.2.840.113556.1.4.800 - Active directory - Feature - MICROSOFT
  Supported SASL mechanisms: 
    GSSAPI, GSS-SPNEGO, EXTERNAL, DIGEST-MD5
  Schema entry: 
    CN=Aggregate,CN=Schema,CN=Configuration,DC=working,DC=example,DC=com
Other:
  currentTime: 
    20200325135056.0Z
  dsServiceName: 
    CN=NTDS Settings,CN=AD,CN=Servers,CN=Sites,CN=Configuration,DC=working,DC=example,DC=com
  defaultNamingContext: 
    DC=working,DC=example,DC=com
  schemaNamingContext: 
    CN=Schema,CN=Configuration,DC=working,DC=example,DC=com
  configurationNamingContext: 
    CN=Configuration,DC=working,DC=example,DC=com
  rootDomainNamingContext: 
    DC=working,DC=example,DC=com
  supportedLDAPPolicies: 
    MaxPoolThreads
    MaxPercentDirSyncRequests
    MaxDatagramRecv
    MaxReceiveBuffer
    InitRecvTimeout
    MaxConnections
    MaxConnIdleTime
    MaxPageSize
    MaxBatchReturnMessages
    MaxQueryDuration
    MaxTempTableSize
    MaxResultSetSize
    MinResultSets
    MaxResultSetsPerConn
    MaxNotificationPerConn
    MaxValRange
    MaxValRangeTransitive
    ThreadMemoryLimit
    SystemMemoryLimitPercent
  highestCommittedUSN: 
    25304263
  dnsHostName: 
ad.working.example.com
  ldapServiceName: 
    working.example.com:ad$@WOKGING.EXAMPLE.COM
  isSynchronized:
    TRUE
  isGlobalCatalogReady: 
    TRUE
  domainFunctionality: 
    4
  forestFunctionality: 
    4
  domainControllerFunctionality: 
    6

pagination not working on this AD REALM with bonsai:

DSA info (from DSE):
  Supported LDAP versions: 3, 2
  Naming contexts: 
    DC=nonworking,DC=example,DC=com
    CN=Configuration,DC=nonworking,DC=example,DC=com
    CN=Schema,CN=Configuration,DC=nonworking,DC=example,DC=com
  Supported controls: 
    1.2.840.113556.1.4.1338 - Verify name - Control - MICROSOFT
    1.2.840.113556.1.4.1339 - Domain scope - Control - MICROSOFT
    1.2.840.113556.1.4.1340 - Search options - Control - MICROSOFT
    1.2.840.113556.1.4.1341 - RODC DCPROMO - Control - MICROSOFT
    1.2.840.113556.1.4.1413 - Permissive modify - Control - MICROSOFT
    1.2.840.113556.1.4.1504 - Attribute scoped query - Control - MICROSOFT
    1.2.840.113556.1.4.1852 - User quota - Control - MICROSOFT
    1.2.840.113556.1.4.1907 - Server shutdown notify - Control - MICROSOFT
    1.2.840.113556.1.4.1948 - Range retrieval no error - Control - MICROSOFT
    1.2.840.113556.1.4.1974 - Server force update - Control - MICROSOFT
    1.2.840.113556.1.4.2026 - Input DN - Control - MICROSOFT
    1.2.840.113556.1.4.2064 - Show recycled - Control - MICROSOFT
    1.2.840.113556.1.4.2065 - Show deactivated link - Control - MICROSOFT
    1.2.840.113556.1.4.2066 - Policy hints [DEPRECATED] - Control - MICROSOFT
    1.2.840.113556.1.4.2090 - DirSync EX - Control - MICROSOFT
    1.2.840.113556.1.4.2204 - Tree deleted EX - Control - MICROSOFT
    1.2.840.113556.1.4.2205 - Updates stats - Control - MICROSOFT
    1.2.840.113556.1.4.2206 - Search hints - Control - MICROSOFT
    1.2.840.113556.1.4.2211 - Expected entry count - Control - MICROSOFT
    1.2.840.113556.1.4.2239 - Policy hints - Control - MICROSOFT
    1.2.840.113556.1.4.2255 - Set owner - Control - MICROSOFT
    1.2.840.113556.1.4.2256 - Bypass quota - Control - MICROSOFT
    1.2.840.113556.1.4.2309
    1.2.840.113556.1.4.319 - LDAP Simple Paged Results - Control - RFC2696
    1.2.840.113556.1.4.417 - LDAP server show deleted objects - Control - MICROSOFT
    1.2.840.113556.1.4.473 - Sort Request - Control - RFC2891
    1.2.840.113556.1.4.474 - Sort Response - Control - RFC2891
    1.2.840.113556.1.4.521 - Cross-domain move - Control - MICROSOFT
    1.2.840.113556.1.4.528 - Server search notification - Control - MICROSOFT
    1.2.840.113556.1.4.529 - Extended DN - Control - MICROSOFT
    1.2.840.113556.1.4.619 - Lazy commit - Control - MICROSOFT
    1.2.840.113556.1.4.801 - Security descriptor flags - Control - MICROSOFT
    1.2.840.113556.1.4.802 - Range option - Control - MICROSOFT
    1.2.840.113556.1.4.805 - Tree delete - Control - MICROSOFT
    1.2.840.113556.1.4.841 - Directory synchronization - Control - MICROSOFT
    1.2.840.113556.1.4.970 - Get stats - Control - MICROSOFT
  Supported extensions: 
    1.2.840.113556.1.4.1781 - Fast concurrent bind - Extension - MICROSOFT
    1.2.840.113556.1.4.2212 - Batch request - Extension - MICROSOFT
    1.3.6.1.4.1.1466.101.119.1 - Dynamic Refresh - Extension - RFC2589
    1.3.6.1.4.1.1466.20037 - StartTLS - Extension - RFC4511-RFC4513
    1.3.6.1.4.1.4203.1.11.3 - Who am I - Extension - RFC4532
  Supported features: 
    1.2.840.113556.1.4.1670 - Active directory V51 - Feature - MICROSOFT
    1.2.840.113556.1.4.1791 - Active directory LDAP Integration - Feature - MICROSOFT
    1.2.840.113556.1.4.1935 - Active directory V60 - Feature - MICROSOFT
    1.2.840.113556.1.4.2080 - Active directory V61 R2 - Feature - MICROSOFT
    1.2.840.113556.1.4.2237 - Active directory W8 - Feature - MICROSOFT
    1.2.840.113556.1.4.800 - Active directory - Feature - MICROSOFT
  Supported SASL mechanisms: 
    GSSAPI, GSS-SPNEGO, EXTERNAL, DIGEST-MD5
  Schema entry: 
    CN=Aggregate,CN=Schema,CN=Configuration,DC=nonworking,DC=example,DC=com
Other:
  currentTime: 
    20200325135058.0Z
  dsServiceName: 
    CN=NTDS Settings,CN=AD,CN=Servers,CN=Sites,CN=Configuration,DC=nonworking,DC=example,DC=com
  defaultNamingContext: 
    DC=nonworking,DC=example,DC=com
  schemaNamingContext: 
    CN=Schema,CN=Configuration,DC=nonworking,DC=example,DC=com
  configurationNamingContext: 
    CN=Configuration,DC=nonworking,DC=example,DC=com
  rootDomainNamingContext: 
    DC=nonworking,DC=example,DC=com
  supportedLDAPPolicies: 
    MaxPoolThreads
    MaxPercentDirSyncRequests
    MaxDatagramRecv
    MaxReceiveBuffer
    InitRecvTimeout
    MaxConnections
    MaxConnIdleTime
    MaxPageSize
    MaxBatchReturnMessages
    MaxQueryDuration
    MaxDirSyncDuration
    MaxTempTableSize
    MaxResultSetSize
    MinResultSets
    MaxResultSetsPerConn
    MaxNotificationPerConn
    MaxValRange
    MaxValRangeTransitive
    ThreadMemoryLimit
    SystemMemoryLimitPercent
  highestCommittedUSN: 
    3690393
  dnsHostName: 
ad.nonworking.example.com
  ldapServiceName: 
    nonworking.example.com:ad$@NONWORKING.EXAMPLE.COM
  isSynchronized:
    TRUE
  isGlobalCatalogReady: 
    TRUE
  domainFunctionality: 
    4
  forestFunctionality: 
    4
  domainControllerFunctionality: 
    7
noirello commented 4 years ago

I checked that it might have something to do about checking the page cookie. The validation was wrong in the previous version. I created a test case with 65535 entries collecting them by 128 per page, but I was able to collect every entry.

It's a long shot, but could you try turning off the server chase referral setting before the search?

client = bonsai.LDAPClient('ldap://dc.example.com')
client.set_credentials('SIMPLE', user='user@example.com', password='password')

client.set_server_chase_referrals(False) # No auto referral chasing by the server 

conn = client.connect()
...
schlitzered commented 4 years ago

setting client.set_server_chase_referrals(False) gives me more then "page_size=1000" results, the number of returned items looks valid.

noirello commented 4 years ago

That sounds great. I'll make some adjustments to turn off referral chasing automatically before a paged search starts.