Open nzig opened 3 years ago
hi @nzig ! this should be handled automatically. Connection
objects possess an auto_referral
property that defaults to True
, and referrals are handled
if responses[-2]['result'] == RESULT_REFERRAL:
if self.connection.usage:
self.connection._usage.referrals_received += 1
if self.connection.auto_referrals:
ref_response, ref_result = self.do_operation_on_referral(self._outstanding[message_id], responses[-2]['referrals'])
if ref_response is not None:
responses = ref_response + [ref_result]
responses.append(RESPONSE_COMPLETE)
elif ref_result is not None:
responses = [ref_result, RESPONSE_COMPLETE]
self._referrals = []
could you try to set collect_usage=True
when creating your connection? that should let you check connection._usage.referrals_received
to see if the referrals were received. you can also look at connection._usage.referrals_connections
to see if referral connections got created properly
one possibility is that the referral is being handled, but it doesn't resolve to anything or the library is failing to automatically cross-bind. Active Directory will attempt to construct referrals irregardless of the client and even when it isn't sure if the referral it provides works.
the ldap3
library has some extensive logic for picking out valid referrals and then creating connections to chase them. but it's possible something in that process is failing
def create_referral_connection(self, referrals):
referral_connection = None
selected_referral = None
cachekey = None
valid_referral_list = self.valid_referral_list(referrals)
if valid_referral_list:
preferred_referral_list = [referral for referral in valid_referral_list if
referral['ssl'] == self.connection.server.ssl]
selected_referral = choice(preferred_referral_list) if preferred_referral_list else choice(
valid_referral_list)
cachekey = (selected_referral['host'], selected_referral['port'] or self.connection.server.port, selected_referral['ssl'])
if self.connection.use_referral_cache and cachekey in self.referral_cache:
referral_connection = self.referral_cache[cachekey]
else:
referral_server = Server(host=selected_referral['host'],
port=selected_referral['port'] or self.connection.server.port,
use_ssl=selected_referral['ssl'],
get_info=self.connection.server.get_info,
formatter=self.connection.server.custom_formatter,
connect_timeout=self.connection.server.connect_timeout,
mode=self.connection.server.mode,
allowed_referral_hosts=self.connection.server.allowed_referral_hosts,
tls=Tls(local_private_key_file=self.connection.server.tls.private_key_file,
local_certificate_file=self.connection.server.tls.certificate_file,
validate=self.connection.server.tls.validate,
version=self.connection.server.tls.version,
ca_certs_file=self.connection.server.tls.ca_certs_file) if
selected_referral['ssl'] else None)
from ..core.connection import Connection
referral_connection = Connection(server=referral_server,
user=self.connection.user if not selected_referral['anonymousBindOnly'] else None,
password=self.connection.password if not selected_referral['anonymousBindOnly'] else None,
version=self.connection.version,
authentication=self.connection.authentication if not selected_referral['anonymousBindOnly'] else ANONYMOUS,
client_strategy=SYNC,
auto_referrals=True,
read_only=self.connection.read_only,
check_names=self.connection.check_names,
raise_exceptions=self.connection.raise_exceptions,
fast_decoder=self.connection.fast_decoder,
receive_timeout=self.connection.receive_timeout,
sasl_mechanism=self.connection.sasl_mechanism,
sasl_credentials=self.connection.sasl_credentials)
if self.connection.usage:
self.connection._usage.referrals_connections += 1
referral_connection.open()
...
so it may be that the library didn't chase the referral properly due to a failure to resolve the addresses, or something like authentication/ssl issues with the referral servers (as you can see it tries to reuse the same settings for those on referral unless the referral specifies otherwise).
multi-domain trust via referrals is one of the more convoluted pieces of the LDAP protocol, because it relies on the servers to figure out how to accept incoming requests via referral, and that leaves a lot of room for hard to debug things.
hopefully the connection usage information above can help us chase down the problem (e.g. maybe we see that we received the referrals but failed to set up connections, and go from there). you can also try turning on extended logging for the library.
After running
conn.search(server.info.other["defaultNamingContext"][0], "(objectClass=Domain)")
conn.response
conn._usage
is
Connection Usage:
Time: [elapsed: 0:00:04.418103]
Initial start time: 2021-04-04T09:42:40.672638
Open socket time: 2021-04-04T09:42:40.672638
Last transmitted time: 2021-04-04T09:42:44.892581
Last received time: 2021-04-04T09:42:45.077932
Close socket time:
Server:
Servers from pool: 0
Sockets open: 1
Sockets closed: 0
Sockets wrapped: 1
Bytes: 2098982
Transmitted: 1058
Received: 2097924
Messages: 21
Transmitted: 6
Received: 15
Operations: 6
Abandon: 0
Bind: 3
Add: 0
Compare: 0
Delete: 0
Extended: 0
Modify: 0
ModifyDn: 0
Search: 3
Unbind: 0
Referrals:
Received: 0
Followed: 0
Connections: 0
Restartable tries: 0
Failed restarts: 0
Successful restarts: 0
I also ran the search with ldap3.utils.log.set_library_log_detail_level(ldap3.utils.log.EXTENDED)
and got the following log (again anonymized, and I apologize if I removed something important. 0.0.0.0
is not the real IP of course).
DEBUG:ldap3:ERROR:detail level set to EXTENDED
DEBUG:ldap3:BASIC:start SEARCH operation via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:EXTENDED:search base sanitized to <DC=exmaple,DC=com> for SEARCH operation via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:PROTOCOL:SEARCH request <{'base': 'DC=exmaple,DC=com', 'scope': 2, 'dereferenceAlias': 3, 'sizeLimit': 0, 'timeLimit': 0, 'typesOnly': False, 'filter': '(objectClass=Domain)', 'attributes': ['1.1']}> sent via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:PROTOCOL:new message id <7> generated
DEBUG:ldap3:NETWORK:sending 1 ldap message for <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:EXTENDED:ldap message sent via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>:
>>LDAPMessage:
>> messageID=7
>> protocolOp=ProtocolOp:
>> searchRequest=SearchRequest:
>> baseObject=DC=exmaple,DC=com
>> scope=wholeSubtree
>> derefAliases=derefAlways
>> sizeLimit=0
>> timeLimit=0
>> typesOnly=False
>> filter=Filter:
>> equalityMatch=EqualityMatch:
>> attributeDesc=objectClass
>> assertionValue=Domain
>> attributes=AttributeSelection:
>> 1.1
DEBUG:ldap3:NETWORK:sent 73 bytes via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:NETWORK:received 42 bytes via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:NETWORK:received 69 bytes via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:NETWORK:received 67 bytes via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:NETWORK:received 71 bytes via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:NETWORK:received 91 bytes via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:NETWORK:received 91 bytes via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:NETWORK:received 75 bytes via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:NETWORK:received 22 bytes via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:NETWORK:received 8 ldap messages via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:EXTENDED:ldap message received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>:
<<{'controls': None,
<< 'messageID': 7,
<< 'payload': [(0, False, 4, b'DC=exmaple,DC=com'), (0, True, 16, [])],
<< 'protocolOp': 4}
DEBUG:ldap3:EXTENDED:ldap message received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>:
<<{'controls': None,
<< 'messageID': 7,
<< 'payload': [(0,
<< False,
<< 4,
<< b'ldaps://sub1.exmaple.com/DC=sub1,DC=exmaple,DC=com')],
<< 'protocolOp': 19}
DEBUG:ldap3:EXTENDED:ldap message received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>:
<<{'controls': None,
<< 'messageID': 7,
<< 'payload': [(0,
<< False,
<< 4,
<< b'ldaps://sub2.exmaple.com/DC=sub2,DC=exmaple,DC=com')],
<< 'protocolOp': 19}
DEBUG:ldap3:EXTENDED:ldap message received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>:
<<{'controls': None,
<< 'messageID': 7,
<< 'payload': [(0,
<< False,
<< 4,
<< b'ldaps://sub3.exmaple.com/DC=sub3,DC=exmaple,DC=com')],
<< 'protocolOp': 19}
DEBUG:ldap3:EXTENDED:ldap message received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>:
<<{'controls': None,
<< 'messageID': 7,
<< 'payload': [(0,
<< False,
<< 4,
<< b'ldaps://ForestDnsZones.exmaple.com/DC=ForestDnsZones,DC=ut'
<< b'houston,DC=com')],
<< 'protocolOp': 19}
DEBUG:ldap3:EXTENDED:ldap message received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>:
<<{'controls': None,
<< 'messageID': 7,
<< 'payload': [(0,
<< False,
<< 4,
<< b'ldaps://DomainDnsZones.exmaple.com/DC=DomainDnsZones,DC=ut'
<< b'houston,DC=com')],
<< 'protocolOp': 19}
DEBUG:ldap3:EXTENDED:ldap message received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>:
<<{'controls': None,
<< 'messageID': 7,
<< 'payload': [(0,
<< False,
<< 4,
<< b'ldaps://exmaple.com/CN=Configuration,DC=exmaple,DC=com')],
<< 'protocolOp': 19}
DEBUG:ldap3:EXTENDED:ldap message received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>:
<<{'controls': None,
<< 'messageID': 7,
<< 'payload': [(0, False, 10, 0), (0, False, 4, b''), (0, False, 4, b'')],
<< 'protocolOp': 5}
DEBUG:ldap3:PROTOCOL:SEARCH response entry <{'raw_dn': b'DC=exmaple,DC=com', 'dn': 'DC=exmaple,DC=com', 'raw_attributes': {}, 'attributes': {}, 'type': 'searchResEntry'}> received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:PROTOCOL:SEARCH response reference <{'uri': ['ldaps://sub1.exmaple.com/DC=sub1,DC=exmaple,DC=com'], 'type': 'searchResRef'}> received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:PROTOCOL:SEARCH response reference <{'uri': ['ldaps://sub2.exmaple.com/DC=sub2,DC=exmaple,DC=com'], 'type': 'searchResRef'}> received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:PROTOCOL:SEARCH response reference <{'uri': ['ldaps://sub3.exmaple.com/DC=sub3,DC=exmaple,DC=com'], 'type': 'searchResRef'}> received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:PROTOCOL:SEARCH response reference <{'uri': ['ldaps://ForestDnsZones.exmaple.com/DC=ForestDnsZones,DC=exmaple,DC=com'], 'type': 'searchResRef'}> received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:PROTOCOL:SEARCH response reference <{'uri': ['ldaps://DomainDnsZones.exmaple.com/DC=DomainDnsZones,DC=exmaple,DC=com'], 'type': 'searchResRef'}> received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:PROTOCOL:SEARCH response reference <{'uri': ['ldaps://exmaple.com/CN=Configuration,DC=exmaple,DC=com'], 'type': 'searchResRef'}> received via <ldaps://ldap.exmaple.com:636 - ssl - user: \svc-user - not lazy - bound - open - <local: 127.0.0.1:43961 - remote: 0.0.0.0:636> - tls not started - listening - RestartableStrategy - internal decoder>
DEBUG:ldap3:BASIC:done SEARCH operation, result <True>
hi @nzig !
thanks for the details! they're incredibly helpful. I've done a little bit of digging and some of what I said earlier was incomplete information. in the LDAP protocol, a search result reference is notedly different from a referral.
in the case of a referral, no true results are returned and a client is routed to another server to perform the same search. in the case of a search result reference, it is simply an element of a search's results, and points directly to an object in another server. a response that contains references uses the same LDAP return code as a response that doesn't, whereas referrals have their own response code
the behavior of the ldap3 library allows automatic following of referrals, but doesn't include any behavior for automatically following search result references. one of the tricky pieces of references is that you're not guaranteed to find a final result at them. referrals say "go find this thing here" whereas references say "go and run this same search here", and in the case of subtree searches like the one you're running this can produce even more references.
so blind reference following can have a growth factor to it. right now the ldap3 library doesn't support reference following. I can poke around at adding it, but probably I'll check in with others to get their thoughts on allowing limits to be set on reference depth and such.
for now, your workaround seems like it will need to continue, as the library doesn't automatically chase references, only referrals.
Thanks for your efforts! In the meantime, I suggest changing the documentation from
searchResRef: the response is a continuation referral to another DIT where the search should continue (usually handled automatically by ldap3)
to something like:
searchResRef: the response is a continuation referral to another DIT where the search should continue (should be handled by the library consumer if required)
I'm searching in Active Directory with ldap3, and it appears that the AD server is returning
searchResRef
referrals that are not being followed. I'm using this (anonymized)which returns (also anonymized)
If I do the same thing but connect to
sub1.example.com
, I will getRFC5411 says that
and the ldap3 docs say
However this does not appear to be the case. My current solution is to connect to all the servers in the
searchResRef
s and run the search there as well. Is there a way for ldap3 to do this automatically?