search

denshoproject / ddr-local

Web UI used for interacting with DDR collections and entities on a local machine.
Other
3 stars 0 forks source link

export of object/file .csv from webui fails with 500 internal error #287

Open pkikawa opened 4 years ago

pkikawa commented 4 years ago

Error at _ui_collection_ddr-densho-399-objects.csv.zip

tested in 5.0.0, 5.0.4 and 5.0.6 (html and logs are from the 5.0.6) on local entiat. local and celeryd.zip

Edit: also present in 5.0.9

gjost commented 4 years ago 
Internal Server Error: /ui/collection/ddr-testing-40359-objects.csv
Traceback (most recent call last):
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/ddr-local/ddrlocal/storage/decorators.py", line 83, in inner
    return func(request, *args, **kwargs)
  File "/opt/ddr-local/ddrlocal/webui/views/collections.py", line 455, in csv_download
    for row in reader:
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
GeoffFroh commented 4 years ago

Looks like the issue is due to changes in Python 3 csv handling and/or custom csv read functionality in the view. Probably needs a re-factor.

pkikawa commented 4 years ago

objcsv509.zip

error on 5.0.9 objects export from ui

gjost commented 4 years ago

Fixed in ddr-local commit 49a45fd for next package.

pkikawa commented 4 years ago

still failing in 5.0.11 objexportfailure.zip

tested on testing collection on 5.0.11 on entait workstation. clicked export objects link in webui.

pkikawa commented 4 years ago

Traceback (most recent call last): File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/celery/app/trace.py", line 480, in trace_task uuid, retval, task_request, publish_result, File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/celery/backends/base.py", line 158, in mark_as_done self.store_result(task_id, result, state, request=request) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/celery/backends/base.py", line 443, in store_result request=request, kwargs) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/celery/backends/base.py", line 861, in _store_result self._set_with_state(self.get_key_for_task(taskid), self.encode(meta), state) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/celery/backends/base.py", line 332, in encode , _, payload = self._encode(data) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/celery/backends/base.py", line 336, in _encode return dumps(data, serializer=self.serializer) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/usr/lib/python3.7/contextlib.py", line 130, in exit self.gen.throw(type, value, traceback) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/kombu/serialization.py", line 54, in _reraise_errors reraise(wrapper, wrapper(exc), sys.exc_info()[2]) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/vine/five.py", line 194, in reraise raise value.with_traceback(tb) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/kombu/serialization.py", line 50, in _reraise_errors yield File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/kombu/utils/json.py", line 70, in dumps dict(default_kwargs, kwargs)) File "/usr/lib/python3.7/json/init.py", line 238, in dumps kw).encode(obj) File "/usr/lib/python3.7/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/kombu/utils/json.py", line 59, in default return super(JSONEncoder, self).default(o) File "/usr/lib/python3.7/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.class.name} ' kombu.exceptions.EncodeError: Object of type PosixPath is not JSON serializable

gjost commented 4 years ago

Check the various places where tasks are sent to Celery AND where responses are sent back to Celery. Check for non-str values, especially for pathlib.Path objects.

Seealso https://github.com/denshoproject/ddr-local/issues/289