Getting <h1>Server Error (500)</h1> from api/dirtree on non-ascii folders

[guysoft]

I can't see the directory tree in /data when trying to add a folder to scan.

I can reproduce this by running in browser or in curl:

guy@golem4:~$ curl 'https://pics.gnethomelinux.com/api/dirtree/' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0' -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.7,he;q=0.3' --compressed -H 'Referer: https://pics.gnethomelinux.com/admin' -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNTU2Mzk5MzYyLCJqdGkiOiI2YTU3OWJmZTFiYzk0OGQ4OTZhYjU1ZDQ2YzE5OTdhMiIsInVzZXJfaWQiOjEsIm5hbWUiOiJhZG1pbiIsImlzX2FkbWluIjp0cnVlLCJmaXJzdF9uYW1lIjoiIiwibGFzdF9uYW1lIjoiIiwic2Nhbl9kaXJlY3RvcnkiOiIiLCJuZXh0Y2xvdWRfc2VydmVyX2FkZHJlc3MiOm51bGwsIm5leHRjbG91ZF91c2VybmFtZSI6bnVsbH0.LD68ExLbczpTieEdqctXL_3I_Ao5tYRfz73oWYf_9_o' -H 'Connection: keep-alive' -H 'Cookie: __utma=110183123.2027832343.1555191015.1555315907.1556300497.5; __utmz=110183123.1555191015.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); csrftoken=GVLYBjnizbHmH3v56uQSgYa3nmdzKEOZwdvdzbiUJA7rskgsyEt0rUkijhSvO4qC; jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNTU2Mzk5MzYyLCJqdGkiOiI2YTU3OWJmZTFiYzk0OGQ4OTZhYjU1ZDQ2YzE5OTdhMiIsInVzZXJfaWQiOjEsIm5hbWUiOiJhZG1pbiIsImlzX2FkbWluIjp0cnVlLCJmaXJzdF9uYW1lIjoiIiwibGFzdF9uYW1lIjoiIiwic2Nhbl9kaXJlY3RvcnkiOiIiLCJuZXh0Y2xvdWRfc2VydmVyX2FkZHJlc3MiOm51bGwsIm5leHRjbG91ZF91c2VybmFtZSI6bnVsbH0.LD68ExLbczpTieEdqctXL_3I_Ao5tYRfz73oWYf_9_o; test=obtain' -H 'TE: Trailers'
<h1>Server Error (500)</h1>

From ownphotos logs I am getting:

2019-04-27 21:04:36,607 : views.py : get : 1603 : INFO : about to get root path tree
2019-04-27 21:04:36,796 : views.py : get : 1605 : INFO : root path tree calculated

[guysoft]

In debug mode when runnign curl and tail I get this:

==> log/ownphotos.log <==
2019-04-27 21:33:45,743 : views.py : get : 1603 : INFO : about to get root path tree
2019-04-27 21:33:45,935 : views.py : get : 1605 : INFO : root path tree calculated

==> log/gunicorn_django.log <==
[2019-04-27 21:33:54 +0000] [146] [ERROR] Error handling request /api/dirtree/
Traceback (most recent call last):
  File "/miniconda/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/miniconda/lib/python3.6/site-packages/django/core/handlers/base.py", line 156, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/miniconda/lib/python3.6/site-packages/django/core/handlers/base.py", line 154, in _get_response
    response = response.render()
  File "/miniconda/lib/python3.6/site-packages/django/template/response.py", line 106, in render
    self.content = self.rendered_content
  File "/miniconda/lib/python3.6/site-packages/rest_framework/response.py", line 72, in rendered_content
    ret = renderer.render(self.data, accepted_media_type, context)
  File "/miniconda/lib/python3.6/site-packages/rest_framework/renderers.py", line 118, in render
    return bytes(ret.encode('utf-8'))
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 31996-32003: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/code/api/middleware.py", line 9, in __call__
    response = self.get_response(request)
  File "/miniconda/lib/python3.6/site-packages/django/core/handlers/exception.py", line 36, in inner
    response = response_for_exception(request, exc)
  File "/miniconda/lib/python3.6/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/miniconda/lib/python3.6/site-packages/django/core/handlers/exception.py", line 125, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/miniconda/lib/python3.6/site-packages/django/views/debug.py", line 95, in technical_500_response
    return HttpResponse(html, status=status_code, content_type='text/html')
  File "/miniconda/lib/python3.6/site-packages/django/http/response.py", line 291, in __init__
    self.content = content
  File "/miniconda/lib/python3.6/site-packages/django/http/response.py", line 321, in content
    content = self.make_bytes(value)
  File "/miniconda/lib/python3.6/site-packages/django/http/response.py", line 236, in make_bytes
    return bytes(value.encode(self.charset))
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 6386-6393: surrogates not allowed

[guysoft]

Ok, error seems to be in https://github.com/hooram/ownphotos/blob/dev/api/middleware.py#L5

[Ramblurr]

I am getting this error too. I have many photo files/directory name with non ascii characters (in German). Can't get going with ownphotos because of this bug :/

[dyyd]

Seems that I have stumbled upon this as well. Any suggestions on how to fix this?

[guysoft]

@hooram is not maintaining this anymore, I gave you the path in the code where it needs to be fixed. The suggestion I have is you either fix the code yourself of find someone who knows how.

[dyyd]

If @hooram is not maintaining it, is anyone? Should I make a pull request or just fork the whole project?

[guysoft]

I can pull request stuff since I have permissions. And I will if you have a solution.

You are also welcome to fork if that makes sense for you.