search

denshoproject / ddr-local

Web UI used for interacting with DDR collections and entities on a local machine.
Other
3 stars 0 forks source link

JSONDecodeError in collection.children #255

Closed gjost closed 5 years ago

gjost commented 5 years ago

@dana Getting this error: JSONDecodeError at /ui/collection/ddr-densho-1000/children/ when I click on the objects tab for densho-1000 (local version). Also if I navigate directly to an interview record and try to add a new segment under the children tab, I get this: JSONDecodeError at /ui/entity/ddr-densho-1000-452/new-idservice/. Tried this about 5 times for ddr-densho-1000-452, and the new segments are not visible, so I'm not sure if they've actually been created or not. @pkikawa on Dana's local densho-1000 comp. (v2.7.1MA) Densho-1000 in the idservice is missing a ton of IDs. densho-1000-1 through densho-1000-3 are present. densho-1000-440 through densho-1000-474 are present. no IDs between 3 and 440 are present. not all entity IDs that are present have their correct child entities present in the ID service (densho-100-442 is missing it's child entities) This error thrown from adding a child entity Also happens on /ui/collection/ddr-densho-1000/children/

2019-04-18 12:11:07,475 DEBUG    [decorators.wrapper]  ========================================================================
2019-04-18 12:11:07,475 DEBUG    [decorators.wrapper]  GET /ui/entity/ddr-densho-1000-452/new-idservice/
...
2019-04-18 12:11:16,195 ERROR    [exception.handle_uncaught_exception]  Internal Server Error: /ui/entity/ddr-densho-1000-452/new-idservice/
Traceback (most recent call last):
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/ddr-local/ddrlocal/webui/decorators.py", line 24, in wrapper
    return f(*args, **kwargs)
  File "/opt/ddr-local/ddrlocal/webui/views/decorators.py", line 19, in inner
    return func(request, *args, **kwargs)
  File "/opt/ddr-local/ddrlocal/storage/decorators.py", line 84, in inner
    return func(request, *args, **kwargs)
  File "/opt/ddr-local/ddrlocal/webui/views/entities.py", line 408, in new_idservice
    entity = _create_entity(request, eidentifier, collection, git_name, git_mail)
  File "/opt/ddr-local/ddrlocal/webui/views/entities.py", line 337, in _create_entity
    exit,status = Entity.new(eidentifier, git_name, git_mail, agent=settings.AGENT)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/models/entity.py", line 331, in new
    agent=agent
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/commands.py", line 51, in wrapper
    return f(*args, **kwargs)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/commands.py", line 41, in wrapper
    return f(*args, **kwargs)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/commands.py", line 488, in entity_create
    ccontrol.update_checksums(collection)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/control.py", line 61, in update_checksums
    [ids.append(entity.id) for entity in collection.children()]
  File "/opt/ddr-local/ddrlocal/webui/models.py", line 283, in children
    kids = super(Collection, self).children(quick=True)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/models/collection.py", line 301, in children
    e.signature_id = json.loads('{%s}' % line)['signature_id']
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/simplejson/__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 55 (char 54)

May or may not be related: @pkikawa Densho-1000 in the idservice is missing a ton of IDs. densho-1000-1 through densho-1000-3 are present. densho-1000-440 through densho-1000-474 are present. no IDs between 3 and 440 are present. not all entity IDs that are present have their correct child entities present in the ID service (densho-100-442 is missing it's child entities) Dana says this behavior started sometime after 3pm yesterday

gjost commented 5 years ago

The error occurs in DDR.models.collection.Collection.children in line 301. It looks like ddr-densho-1000/files/ddr-densho-1000-452/entity.json is unusual in having a child/segment record containing a signature_id while the main entity does not have a signature_id.

The children function is trying to be a bit too smart, to optimize by manually reading the JSON file instead of just loading the whole thing and parsing it. It tries to load title and signature_id lines as miniature JSON documents, and quits once it has both a title and a signature_id. In this case it doesn't have a signature_id until it's gotten into the child entities, and this line has a trailing comma and does not parse as legal JSON, triggering this error.

gjost commented 5 years ago

So basically this happens when you have an Entity without a signature, that has a segment with a signature.

gjost commented 5 years ago

This is actually a ddr-cmdln issue: https://github.com/densho/ddr-cmdln/issues/142