_search method throws uncaught KeyError

[bonzani]

When I use the the example configuration and adapt it to our LDAP server, I always get an KeyError on line 159 in main.py.

Traceback (most recent call last):
  File "/opt/devpi/local/lib/python2.7/site-packages/waitress/channel.py", line 336, in service
    task.service()
  File "/opt/devpi/local/lib/python2.7/site-packages/waitress/task.py", line 169, in service
    self.execute()
  File "/opt/devpi/local/lib/python2.7/site-packages/waitress/task.py", line 388, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_server/views.py", line 129, in __call__
    return self.app(environ, start_response)
  File "/opt/devpi/local/lib/python2.7/site-packages/pyramid/router.py", line 223, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/opt/devpi/local/lib/python2.7/site-packages/pyramid/router.py", line 198, in invoke_subrequest
    response = handle_request(request)
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_server/views.py", line 152, in request_log_handler
    response = handler(request)
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_server/views.py", line 187, in request_tx_handler
    response = handler(request)
  File "/opt/devpi/local/lib/python2.7/site-packages/pyramid/tweens.py", line 20, in excview_tween
    response = handler(request)
  File "/opt/devpi/local/lib/python2.7/site-packages/pyramid/router.py", line 145, in handle_request
    view_name
  File "/opt/devpi/local/lib/python2.7/site-packages/pyramid/view.py", line 541, in _call_view
    response = view_callable(context, request)
  File "/opt/devpi/local/lib/python2.7/site-packages/pyramid/config/views.py", line 328, in attr_view
    return view(context, request)
  File "/opt/devpi/local/lib/python2.7/site-packages/pyramid/config/views.py", line 304, in predicate_wrapper
    return view(context, request)
  File "/opt/devpi/local/lib/python2.7/site-packages/pyramid/config/views.py", line 385, in viewresult_to_response
    result = view(context, request)
  File "/opt/devpi/local/lib/python2.7/site-packages/pyramid/config/views.py", line 483, in _class_requestonly_view
    response = getattr(inst, attr)()
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_server/views.py", line 915, in login
    proxyauth = self.auth.new_proxy_auth(user, password)
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_server/auth.py", line 85, in new_proxy_auth
    result = self._validate(username, password)
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_server/auth.py", line 47, in _validate
    password=authpassword)
  File "/opt/devpi/local/lib/python2.7/site-packages/pluggy.py", line 724, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/opt/devpi/local/lib/python2.7/site-packages/pluggy.py", line 338, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/opt/devpi/local/lib/python2.7/site-packages/pluggy.py", line 333, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/opt/devpi/local/lib/python2.7/site-packages/pluggy.py", line 596, in execute
    res = hook_impl.function(*args)
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_ldap/main.py", line 239, in devpiserver_auth_user
    return ldap.validate(username, password)
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_ldap/main.py", line 206, in validate
    userdn = self._userdn(username)
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_ldap/main.py", line 183, in _userdn
    result = self._search(None, self['user_search'], username=username)
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_ldap/main.py", line 159, in _search
    return sum((x['attributes'][attribute_name] for x in conn.response), [])
  File "/opt/devpi/local/lib/python2.7/site-packages/devpi_ldap/main.py", line 159, in <genexpr>
    return sum((x['attributes'][attribute_name] for x in conn.response), [])
KeyError: u'attributes'

I investigated the response of the ldap server and found out, that the server retourned additional dictionaries with meta information. My fix was this

if found:
    if any(('attributes' in x and attribute_name in x['attributes']) for x in conn.response):
        def extract_search(s):
            return s['attributes'][attribute_name]
    elif attribute_name in ('dn', 'distinguishedName'):
        def extract_search(s):
            return [s[attribute_name]]
    else:
        threadlog.error('configured attribute_name {} not found in any search results'.format(attribute_name))
        return []
    res = []
    for x in conn.response:
        try:
            res += extract_search(x)
        except KeyError:
            # catch lookup errors
            continue
    return res
else:
    threadlog.error("Search failed %s %s: %s" % (search_filter, config, conn.result))
    return []

I think it would be great if in some stable way the access to the dictionary would be checked or the exception be caught, so that the authentification doesn't fail just because there was additional information.

[fschulze]

Would it be possible to post the additional dicts you get? I guess the are in conn.response? It's ok if you obfuscate sensitive information. I just want to see the structure of the replies.

[bonzani]

Yes, that's correct, they are in conn.response. I extracted the result of the user search. You can see that there are three additional dicts.

[
  {'dn': 'CN=display name,OU=Users,OU=location,OU=COMPANY,DC=company,DC=ch',
   'attributes': {'distinguishedName': [u'CN=display name,OU=Users,OU=location,OU=COMPANY,DC=company,DC=ch']},
   'raw_attributes': {'distinguishedName': ['CN=display name,OU=Users,OU=location,OU=COMPANY,DC=company,DC=ch']},
   'type': 'searchResEntry'
  },
  {'type': 'searchResRef',
   'uri': ['ldap://DomainDnsZones.company.ch/DC=DomainDnsZones,DC=company,DC=ch']
  },
  {'type': 'searchResRef',
   'uri': ['ldap://ForestDnsZones.company.ch/DC=ForestDnsZones,DC=company,DC=ch']
  },
  {'type': 'searchResRef',
   'uri': ['ldap://company.ch/CN=Configuration,DC=company,DC=ch']
  }
]

By the way the same goes for the group search. There are also this last three additional dicts in the response.

[fschulze]

Could you check if #26 fixes your issue?

[bonzani]

Yes this fixes the issue. Thanks very much for the fast change.