Closed fredreichbier closed 6 years ago
Hi, are you using the same pyasn1 version? There have been many version update of pyasn1, are you using the last version (0.4.2)?
Il giorno 24 gen 2018, alle ore 17:15, Friedrich Weber notifications@github.com ha scritto:
First, thanks for the great module! I stumbled upon some weird differing behavior between versions with the following script (running against an Univention Corporate Server, i.e., OpenLDAP) which searches for an entry based on a given entryUUID value.
import ldap3
SERVER = '...' BIND_DN = '...' BIND_PW = '...' BASE_DN = '...' FILTER = '(entryUUID=6ed31028-d30d-1036-9fd8-cd5127ce8b3b)'
server = ldap3.Server(SERVER, get_info=ldap3.ALL) conn = ldap3.Connection(server, BIND_DN, BIND_PW, auto_bind=True) print server.schema.attribute_types['entryUUID'] print conn.search(BASE_DN, FILTER), conn.response Here are the results with different ldap3 versions:
ldap3 2.3: everything works fine
Attribute type: 1.3.6.1.1.16.4 Short name: entryUUID Description: UUID of the entry Single value: True No user modification: True Usage: Directory operation Equality rule: UUIDMatch Ordering rule: UUIDOrderingMatch Syntax: 1.3.6.1.1.16.1 [('1.3.6.1.1.16.1', 'LDAP_SYNTAX', 'Universally Unique Identifier (UUID)', 'RFC4530')] OidInfo: ('1.3.6.1.1.16.4', 'ATTRIBUTE_TYPE', 'entryUUID', 'RFC4530')
True [{'dn': u'uid=user111,cn=users,dc=ucs-test,dc=intranet', 'attributes': {}, 'raw_attributes': {}, 'raw_dn': 'uid=user111,cn=users,dc=ucs-test,dc=intranet', 'type': 'searchResEntry'}] ldap3 2.4: search finds no results
Attribute type: 1.3.6.1.1.16.4 Short name: entryUUID Description: UUID of the entry Single value: True No user modification: True Usage: Directory operation Equality rule: UUIDMatch Ordering rule: UUIDOrderingMatch Syntax: 1.3.6.1.1.16.1 [('1.3.6.1.1.16.1', 'LDAP_SYNTAX', 'Universally Unique Identifier (UUID)', 'RFC4530')] OidInfo: ('1.3.6.1.1.16.4', 'ATTRIBUTE_TYPE', 'entryUUID', 'RFC4530')
False [] ldap 2.4.1: error occurs
Attribute type: 1.3.6.1.1.16.4 Short name: entryUUID Description: UUID of the entry Single value: True No user modification: True Usage: Directory operation Equality rule: UUIDMatch Ordering rule: UUIDOrderingMatch Syntax: 1.3.6.1.1.16.1 [('1.3.6.1.1.16.1', 'LDAP_SYNTAX', 'Universally Unique Identifier (UUID)', 'RFC4530')] OidInfo: ('1.3.6.1.1.16.4', 'ATTRIBUTE_TYPE', 'entryUUID', 'RFC4530')
Traceback (most recent call last): File "search.py", line 12, in
print conn.search(BASE_DN, FILTER), conn.response File "/home/fred/privacyidea/privacyidea/venv/lib/python2.7/site-packages/ldap3/core/connection.py", line 775, in search response = self.post_send_search(self.send('searchRequest', request, controls)) File "/home/fred/privacyidea/privacyidea/venv/lib/python2.7/site-packages/ldap3/strategy/base.py", line 303, in send self.connection.request = BaseStrategy.decode_request(message_type, request, controls) File "/home/fred/privacyidea/privacyidea/venv/lib/python2.7/site-packages/ldap3/strategy/base.py", line 635, in decode_request result = search_request_to_dict(component) File "/home/fred/privacyidea/privacyidea/venv/lib/python2.7/site-packages/ldap3/operation/search.py", line 516, in search_request_to_dict 'filter': filter_to_string(request['filter']), File "/home/fred/privacyidea/privacyidea/venv/lib/python2.7/site-packages/ldap3/operation/search.py", line 474, in filter_to_string ava = ava_to_dict(filter_object['equalityMatch']) File "/home/fred/privacyidea/privacyidea/venv/lib/python2.7/site-packages/ldap3/protocol/convert.py", line 87, in ava_to_dict return {'attribute': str(ava['attributeDesc']), 'value': escape_filter_chars(str(ava['assertionValue']))} File "/home/fred/privacyidea/privacyidea/venv/lib/python2.7/site-packages/ldap3/utils/conv.py", line 91, in escape_filter_chars text = to_unicode(text, encoding) File "/home/fred/privacyidea/privacyidea/venv/lib/python2.7/site-packages/ldap3/utils/conv.py", line 64, in to_unicode raise UnicodeError("Unable to convert client data to unicode: %r" % obj) UnicodeError: Unable to convert client data to unicode: "n\xd3\x10(\xd3\r\x106\x9f\xd8\xcdQ'\xce\x8b;" I had a look at the changelog, but couldn't spot anything which would explain this. Do you maybe have a hint about what is happening here? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
Hi, yes, I'm using pyasn1 0.4.2 in all three cases.
Hi, sorry for messing up with the entryUUID attribute! In version 2.4 I added a more compliant validating scheme for the attribute values. I didn't notiche that RFC4530 states that when searching for entryUUID the filter must include the string representation and not the byte representation of the UUID.
Can you try the code in dev? This should work now.
Hi, thank you for looking into this! I can confirm that the above code works and returns the correct number of results with the current dev branch (7dea53ebed620108f1c5e7a870510fa4ec0c4e7d).
However, with the current dev branch, I still have a similar problem with Active Directory and objectGUID
(again, using pyasn1 0.4.2):
import ldap3
import pyasn1
SERVER = '...'
BIND_DN = 'cn=Administrator,cn=users,dc=testfoo,dc=intranet'
BIND_PW = '...'
BASE_DN = 'dc=testfoo,dc=intranet'
FILTER = '(&(sAMAccountName=*)(objectClass=person)(objectGUID=\\2c\\4f\\20\\73\\49\\b9\\39\\4c\\a3\\ae\\79\\fb\\51\\19\\e6\\76))'
server = ldap3.Server(SERVER, get_info=ldap3.ALL)
conn = ldap3.Connection(server, BIND_DN, BIND_PW, auto_bind=True)
print conn.search(BASE_DN, FILTER), conn.response
ldap3 2.4: works fine
True [{'dn': u'CN=Administrator,CN=Users,DC=testfoo,DC=intranet', 'attributes': {}, 'raw_attributes': {}, 'raw_dn': 'CN=Administrator,CN=Users,DC=testfoo,DC=intranet', 'type': 'searchResEntry'}, {'type': 'searchResRef', 'uri': [u'ldaps://ForestDnsZones.testfoo.intranet/DC=ForestDnsZones,DC=testfoo,DC=intranet']}, {'type': 'searchResRef', 'uri': [u'ldaps://DomainDnsZones.testfoo.intranet/DC=DomainDnsZones,DC=testfoo,DC=intranet']}, {'type': 'searchResRef', 'uri': [u'ldaps://testfoo.intranet/CN=Configuration,DC=testfoo,DC=intranet']}]
ldap3 2.4.1: error
Traceback (most recent call last):
File "/home/fred/privacyidea/privacyidea/search-ad.py", line 11, in <module>
print conn.search(BASE_DN, FILTER), conn.response
File "/tmp/ldap241/lib/python2.7/site-packages/ldap3/core/connection.py", line 775, in search
response = self.post_send_search(self.send('searchRequest', request, controls))
File "/tmp/ldap241/lib/python2.7/site-packages/ldap3/strategy/base.py", line 303, in send
self.connection.request = BaseStrategy.decode_request(message_type, request, controls)
File "/tmp/ldap241/lib/python2.7/site-packages/ldap3/strategy/base.py", line 635, in decode_request
result = search_request_to_dict(component)
File "/tmp/ldap241/lib/python2.7/site-packages/ldap3/operation/search.py", line 516, in search_request_to_dict
'filter': filter_to_string(request['filter']),
File "/tmp/ldap241/lib/python2.7/site-packages/ldap3/operation/search.py", line 466, in filter_to_string
filter_string += filter_to_string(f)
File "/tmp/ldap241/lib/python2.7/site-packages/ldap3/operation/search.py", line 474, in filter_to_string
ava = ava_to_dict(filter_object['equalityMatch'])
File "/tmp/ldap241/lib/python2.7/site-packages/ldap3/protocol/convert.py", line 87, in ava_to_dict
return {'attribute': str(ava['attributeDesc']), 'value': escape_filter_chars(str(ava['assertionValue']))}
File "/tmp/ldap241/lib/python2.7/site-packages/ldap3/utils/conv.py", line 91, in escape_filter_chars
text = to_unicode(text, encoding)
File "/tmp/ldap241/lib/python2.7/site-packages/ldap3/utils/conv.py", line 64, in to_unicode
raise UnicodeError("Unable to convert client data to unicode: %r" % obj)
UnicodeError: Unable to convert client data to unicode: ',O sI\xb99L\xa3\xaey\xfbQ\x19\xe6v'
ldap3 dev:
Traceback (most recent call last):
File "/home/fred/privacyidea/privacyidea/search-ad.py", line 11, in <module>
print conn.search(BASE_DN, FILTER), conn.response
File "/tmp/foo/lib/python2.7/site-packages/ldap3/core/connection.py", line 776, in search
response = self.post_send_search(self.send('searchRequest', request, controls))
File "/tmp/foo/lib/python2.7/site-packages/ldap3/strategy/base.py", line 303, in send
self.connection.request = BaseStrategy.decode_request(message_type, request, controls)
File "/tmp/foo/lib/python2.7/site-packages/ldap3/strategy/base.py", line 635, in decode_request
result = search_request_to_dict(component)
File "/tmp/foo/lib/python2.7/site-packages/ldap3/operation/search.py", line 517, in search_request_to_dict
'filter': filter_to_string(request['filter']),
File "/tmp/foo/lib/python2.7/site-packages/ldap3/operation/search.py", line 467, in filter_to_string
filter_string += filter_to_string(f)
File "/tmp/foo/lib/python2.7/site-packages/ldap3/operation/search.py", line 475, in filter_to_string
ava = ava_to_dict(filter_object['equalityMatch'])
File "/tmp/foo/lib/python2.7/site-packages/ldap3/protocol/convert.py", line 87, in ava_to_dict
return {'attribute': str(ava['attributeDesc']), 'value': escape_filter_chars(str(ava['assertionValue']))}
File "/tmp/foo/lib/python2.7/site-packages/ldap3/utils/conv.py", line 91, in escape_filter_chars
text = to_unicode(text, encoding)
File "/tmp/foo/lib/python2.7/site-packages/ldap3/utils/conv.py", line 64, in to_unicode
raise UnicodeError("Unable to convert client data to unicode: %r" % obj)
UnicodeError: Unable to convert client data to unicode: ',O sI\xb99L\xa3\xaey\xfbQ\x19\xe6v'
Maybe this has a similar reason?
Thank you for looking into this! Is there any way in which I can assist you? (e.g. would it be useful to you to have a reproducible testcase with an accessible AD?)
Sorry for the double-posting, but I ran git-bisect to find the revision after which the objectGUID
search script above starts to crash: Seems like the relevant changes were made in aacdf9ec432d2705148b5d2ae9655fe77ab50164.
I just reproduced this: In its parent revision f58203e, the script works fine, and it gives the error above in revision aacdf9ec432d2705148b5d2ae9655fe77ab50164. Maybe this is useful information?
[crossposting with #504]
Could you please give a try to the code in dev? I've added a validator for the objectGUID, but cannot check it.
Thanks, Giovanni
Hi @cannatag, thanks a lot for looking into this!
Unfortunately, the code above now fails with another exception on d17f34a148c2a2ccef67378291fbe6494fd46997:
Traceback (most recent call last):
File "test.py", line 11, in <module>
print conn.search(BASE_DN, FILTER), conn.response
File "/home/fred/privacyidea/ldap3/ldap3/core/connection.py", line 773, in search
check_names=self.check_names)
File "/home/fred/privacyidea/ldap3/ldap3/operation/search.py", line 372, in search_operation
request['filter'] = compile_filter(parse_filter(search_filter, schema, auto_escape, auto_encode, validator, check_names).elements[0]) # parse the searchFilter string and compile it starting from the root node
File "/home/fred/privacyidea/ldap3/ldap3/operation/search.py", line 206, in parse_filter
current_node.append(evaluate_match(search_filter[start_pos:end_pos], schema, auto_escape, auto_encode, validator, check_names))
File "/home/fred/privacyidea/ldap3/ldap3/operation/search.py", line 166, in evaluate_match
assertion = {'attr': left_part, 'value': validate_assertion_value(schema, left_part, right_part, auto_escape, auto_encode, validator, check_names)}
File "/home/fred/privacyidea/ldap3/ldap3/protocol/convert.py", line 141, in validate_assertion_value
value = validate_attribute_value(schema, name, value, auto_encode, validator=validator, check_names=check_names)
File "/home/fred/privacyidea/ldap3/ldap3/protocol/convert.py", line 162, in validate_attribute_value
raise LDAPInvalidValueError('value \'%s\' non valid for attribute \'%s\'' % (value, name))
ldap3.core.exceptions.LDAPInvalidValueError: value '\2c\4f\20\73\49\b9\39\4c\a3\ae\79\fb\51\19\e6\76' non valid for attribute 'objectGUID'
Apparently, the new validator for objectGUID (validate_uuid_le
) does not like the format in which we specify the objectGUID value in the filter string:
FILTER = '(&(sAMAccountName=*)(objectClass=person)(objectGUID=\\2c\\4f\\20\\73\\49\\b9\\39\\4c\\a3\\ae\\79\\fb\\51\\19\\e6\\76))'
i.e. we specify the objectGUID value as a sequence of backslash-escaped binary values. I think we do that because that was the format in which ldap3 expected objectGUID until now.
But even if I specify the objectGUID in the format expected by validate_uuid_le
:
FILTER = '(&(sAMAccountName=*)(objectClass=person)(objectGUID=73204f2c-b949-4c39-a3ae-79fb5119e676))'
... we again get the UnicodeError
.
Traceback (most recent call last):
File "search-ad.py", line 14, in <module>
print conn.search(BASE_DN, FILTER, auto_escape=False), conn.response
File "/home/fred/privacyidea/ldap3/ldap3/core/connection.py", line 776, in search
response = self.post_send_search(self.send('searchRequest', request, controls))
File "/home/fred/privacyidea/ldap3/ldap3/strategy/base.py", line 303, in send
self.connection.request = BaseStrategy.decode_request(message_type, request, controls)
File "/home/fred/privacyidea/ldap3/ldap3/strategy/base.py", line 635, in decode_request
result = search_request_to_dict(component)
File "/home/fred/privacyidea/ldap3/ldap3/operation/search.py", line 517, in search_request_to_dict
'filter': filter_to_string(request['filter']),
File "/home/fred/privacyidea/ldap3/ldap3/operation/search.py", line 467, in filter_to_string
filter_string += filter_to_string(f)
File "/home/fred/privacyidea/ldap3/ldap3/operation/search.py", line 475, in filter_to_string
ava = ava_to_dict(filter_object['equalityMatch'])
File "/home/fred/privacyidea/ldap3/ldap3/protocol/convert.py", line 89, in ava_to_dict
return {'attribute': str(ava['attributeDesc']), 'value': escape_filter_chars(str(ava['assertionValue']))}
File "/home/fred/privacyidea/ldap3/ldap3/utils/conv.py", line 91, in escape_filter_chars
text = to_unicode(text, encoding)
File "/home/fred/privacyidea/ldap3/ldap3/utils/conv.py", line 64, in to_unicode
raise UnicodeError("Unable to convert client data to unicode: %r" % obj)
UnicodeError: Unable to convert client data to unicode: ',O sI\xb99L\xa3\xaey\xfbQ\x19\xe6v'
Hi, the problem with the objectGUID is that it has different representation against the same byte value (stored in little endian in AD). I've been readying througout the RFC4122 (that defines the UUID standard) and the Microsoft documentation at MS-DTYP: Windows Data Type that defines AD GUID, and I've found that while the objectGUID value is stored always in little endian format in AD, it can be represented in 3 different way:
The data structure for objectGUID is:
typedef struct _GUID {
unsigned long Data1;
unsigned short Data2;
unsigned short Data3;
byte Data4[8];
} GUID,
UUID,
*PGUID;
It can be represented in 3 ways:
So for example the uuid e1e2e3e4-e5e6-e7e8-e9ea-ebecedeeefff
can be represented with:
# string representation, big endian
'{e1e2e3e4-e5e6-e7e8-e9ea-ebecedeeefff}'
# packet representation, little endian
'e4e3e2e1-e6e5-e8e7-e9ea-ebecedeeefff' # packet representation, little endian
# ldap byte representation, little endian
'\\e4\\e3\\e2\\e1\\e6\\e5\\e8\\e7\\e9\\ea\\eb\\ec\\ed\\ee\\ef\\ff'
# python bytes representation, little endian, must be converted with ldap3.utils.conv.escape_bytes() for using it in a filter
bytes('\xe4\xe3\xe2\xe1\xe6\xe5\xe8\xe7\xe9\xea\xeb\xec\xed\xee\xef\xff')
So I've changed the objectGUID validator in ldap3.protocols.formatters.validators.validate_uuid_le()
to accept any of these format and return always the same byte sequence. I've also changed the ldap3.protocols.formatters.formatters.format_uuid_le()
to return the string (with curly braces) representation in big endian.
I'm not sure if this fixes your issue because I've not access to a real AD server now and I've tested it only with the internal ldap3 mock strategies.
Could you please check the code in dev and tell me if my understanding of how AD objectGUID works is correct?
Thanks. Giovanni
Hi Giovanni, first, thanks a lot again for looking into this complex topic! The objectGUID stuff is really a lot more confusing than I anticipated.
I checked the code in dev (6a3131742f13c5873f0deb6de4acd2166a6e790c), and indeed, the AD search script from above works just fine! Moreover, all three alternative formats for specifying the objectGUID work, i.e.
(objectGUID={73204f2c-b949-4c39-a3ae-79fb5119e676})
(objectGUID=\2c\4f\20\73\49\b9\39\4c\a3\ae\79\fb\51\19\e6\76)
(objectGUID=2c4f2073-49b9-394c-a3ae79fb5119e676)
all give the same result. That's great!
I also checked RFC 4122 and MS-DTYP. My two cents:
{73204f2c-b949-4c39-a3ae-79fb5119e676}
) is just fine, as specified in MS-DTYP 2.3.4.3 and the RFC. I also like that format_uuid_le
returns this notation now, as this allows to directly re-use attribute values in filter strings.\2c\4f\20\73\49\b9\39\4c\a3\ae\79\fb\51\19\e6\76
) works as specified.2c4f2073-49b9-394c-a3ae79fb5119e676
). I probably overlooked something, but as this notation and the little-endian LDAP byte notation are pretty much interchangeable, I'm not sure if ldap3 needs to support it?There is only one remaining issue in the current dev
: The entryUUID
script from my initial issue stopped working again and returns no results at all:
Attribute type: 1.3.6.1.1.16.4
Short name: entryUUID
Description: UUID of the entry
Single value: True
No user modification: True
Usage: Directory operation
Equality rule: UUIDMatch
Ordering rule: UUIDOrderingMatch
Syntax: 1.3.6.1.1.16.1 [('1.3.6.1.1.16.1', 'LDAP_SYNTAX', 'Universally Unique Identifier (UUID)', 'RFC4530')]
OidInfo: ('1.3.6.1.1.16.4', 'ATTRIBUTE_TYPE', 'entryUUID', 'RFC4530')
False []
I think this is because the validate_uuid
validator specified for the entryUUID
attribute since 412470f6d converts the value to bytes before sending it, whereas the value should apparently be transmitted as a string. What do you think?
Again, thanks a bunch for your effort!
Best Wishes
Friedrich
You're right. I misinterpreted the packet representation. It should be the string sequence of hex values without the dashes. I will change the validator accordingly:
# string representation, big endian
'{e1e2e3e4-e5e6-e7e8-e9ea-ebecedeeefff}'
# packet representation, little endian
'e4e3e2e1e6e5e8e7e9eaebecedeeefff' # packet representation, little endian
# ldap byte representation, little endian
'\\e4\\e3\\e2\\e1\\e6\\e5\\e8\\e7\\e9\\ea\\eb\\ec\\ed\\ee\\ef\\ff'
# python bytes representation, little endian, must be converted with ldap3.utils.conv.escape_bytes() for using it in a filter
bytes('\xe4\xe3\xe2\xe1\xe6\xe5\xe8\xe7\xe9\xea\xeb\xec\xed\xee\xef\xff')
Should I treat the "just-dashed-without-curly-braces" in big endian as the "dashed-with-curly-braces"? I think this is helpful because not everyone uses curly braces:
# string representation alternative, big endian
'e1e2e3e4-e5e6-e7e8-e9ea-ebecedeeefff'
I've discriminate the Novell GUID from the openLDAP EntryUUID. Now there are two different validator, the Novell GUID needs a byte value for searching, while the openLDAP EntryUUID needs a string representation.
Can you check the code in dev (also for the AD objectGUID)?
Hi Giovanni, I can confirm that on current dev (e681cc6eac2da80113be6131a8f1f6457abbf2a4), searching for entryUUIDs works fine again. The little-endian packet representation without dashes looks fine to me! Also, I find it sensible to make curly braces optional for objectGUID values as you suggested (the Python uuid module also seems to do that).
So, from my point of view, this issue is fixed. :-) Many thanks again for your great help in resolving this!
Ok, will release the fix in the next release of ldap3.
Thanks for your support.
Fixed in v2.5.
First, thanks for the great module! I stumbled upon some weird differing behavior between versions with the following script (running against an Univention Corporate Server, i.e., OpenLDAP) which searches for an entry based on a given
entryUUID
value.Here are the results with different ldap3 versions:
ldap3 2.3: everything works fine
ldap3 2.4: search finds no results
ldap 2.4.1: error occurs
I had a look at the changelog, but couldn't spot anything which would explain this. Do you maybe have a hint about what is happening here?