Failing to add user to organisation

[rossjones]

I get a server error whenever I try to add "[user_d431275]" to https://data.gov.uk/publisher/users/traffic-wales

This appears to be an XML validation error when getting user properties from Drupal. I think this is because of the Welsh chars in their user description.

Stacktrace below:

Module weberror.errormiddleware:162 in __call__
>>  app_iter = self.application(environ, sr_checker)
Module webob.dec:147 in __call__
>>  resp = self.call_func(req, *args, **self.kwargs)
Module webob.dec:208 in call_func
>>  return self.func(req, *args, **kwargs)
Module fanstatic.publisher:234 in __call__
>>  return request.get_response(self.app)
Module webob.request:1053 in get_response
>>  application, catch_exc_info=False)
Module webob.request:1022 in call_application
>>  app_iter = application(self.environ, start_response)
Module webob.dec:147 in __call__
>>  resp = self.call_func(req, *args, **self.kwargs)
Module webob.dec:208 in call_func
>>  return self.func(req, *args, **kwargs)
Module fanstatic.injector:54 in __call__
>>  response = request.get_response(self.app)
Module webob.request:1053 in get_response
>>  application, catch_exc_info=False)
Module webob.request:1022 in call_application
>>  app_iter = application(self.environ, start_response)
Module beaker.middleware:73 in __call__
>>  return self.app(environ, start_response)
Module beaker.middleware:155 in __call__
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:131 in __call__
>>  response = self.app(environ, start_response)
Module ckanext.dgu.authentication.drupal_auth:124 in __call__
>>  return self.app(environ, new_start_response)
Module pylons.wsgiapp:125 in __call__
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
>>  return controller(environ, start_response)
Module ckan.lib.base:347 in __call__
>>  res = WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
>>  return func(**args)
Module ckanext.dgu.controllers.publisher:266 in users
>>  return self._add_users(c.group, request.params)
Module ckanext.dgu.controllers.publisher:205 in _add_users
>>  user_properties = self.drupal_client.get_user_properties(drupal_id)
Module ckanext.dgu.drupalclient:71 in get_user_properties
>>  user = self.drupal.user.retrieve(str(user_id))
Module xmlrpclib:1233 in __call__
>>  return self.__send(self.__name, args)
Module xmlrpclib:1587 in __request
>>  verbose=self.__verbose
Module xmlrpclib:1273 in request
>>  return self.single_request(host, handler, request_body, verbose)
Module xmlrpclib:1306 in single_request
>>  return self.parse_response(response)
Module xmlrpclib:1476 in parse_response
>>  p.feed(data)
Module xmlrpclib:558 in feed
>>  self._parser.Parse(data, 0)
ExpatError: not well-formed (invalid token): line 59, column 133

[davidread]

Is the problem with xmlrpclib or with the response from Drupal? I guess you could dig into the code or use wireshark to see the actual request it makes. I suspect Drupal is returning an error message rather than XML.

[ratajczak]

I can't find anything wrong with Drupal API, this can be tested with curl on prod3:

curl -d '<?xml version="1.0"?><methodCall><methodName>user.retrieve</methodName><params><param><value><string>431275</string></value></param></params></methodCall>' https://data.gov.uk/services/xmlrpc

I've checked the response and it looks like valid XML.

[rossjones]

Looking deeper at this it does come back malformed for a different user (431167)

curl --user AUTH:AUTH -d '<?xml version="1.0"?><methodCall><methodName>user.retrieve</methodName><params><param><value><string>431167</string></value></param></params></methodCall>' https://data.gov.uk/services/xmlrpc

When I run it through xmlvalidation.com it says

The element type "struct" must be terminated by the matching end-tag "</struct>".

[ratajczak]

I can't see any obvious cause, it's probably a bug in a contributed module. I can look at this in depth or alternatively there is REST endpoint available: https://data.gov.uk/services/rest/user/431167 What do you think?

[rossjones]

I'd rather fix it properly if we can. The same user has been having issues with being logged out unexpectedly and it's currently blocking me adding a user who needs to publish soon.

I can fix it locally by mangling our database and manually deleted records, but that isn't ideal because I'm not sure of what any knock on effects might be.

[rossjones]

I've also noted an issue where a user is allowed to enter a vertical tab character (0xb) in their description. When it is added to XML it makes the XML invalid because it is not encoded.

[rossjones]

@ratajczak I've fixed this now by deleting the problematic user from the organisastion, and then removing the vertical tabs from the description of the other user I want to add.

Really we should fix this, but at least we now know what to look for, and should be able to resolve if that happens again - so you may as well leave this until you have free time.