Appresource api chokes on bad unicode in spappresourcedata introduced by Sp6 import

[benanhalt]

uwfc_1                        | [23/Mar/2022 17:15:47] [ERROR] [django.request:228] Internal Server Error: /api/specify/spappresourcedata/
uwfc_1                        | Traceback (most recent call last):
uwfc_1                        |   File "/opt/specify7/ve/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
uwfc_1                        |     response = get_response(request)
uwfc_1                        |   File "/opt/specify7/ve/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
uwfc_1                        |     response = self.process_exception_by_middleware(e, request)
uwfc_1                        |   File "/opt/specify7/ve/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
uwfc_1                        |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
uwfc_1                        |   File "/opt/specify7/specifyweb/specify/views.py", line 20, in wrapped
uwfc_1                        |     return view(request, *args, **kwargs)
uwfc_1                        |   File "/opt/specify7/ve/lib/python3.6/site-packages/django/views/decorators/cache.py", line 31, in _cache_controlled
uwfc_1                        |     response = viewfunc(request, *args, **kw)
uwfc_1                        |   File "/opt/specify7/specifyweb/specify/views.py", line 32, in wrapped
uwfc_1                        |     return view(request, *args, **kwargs)
uwfc_1                        |   File "/opt/specify7/specifyweb/specify/views.py", line 62, in view
uwfc_1                        |     return dispatch_func(request, *args, **kwargs)
uwfc_1                        |   File "/opt/specify7/specifyweb/specify/api.py", line 174, in collection_dispatch
uwfc_1                        |     resp = HttpResponse(toJson(data), content_type='application/json')
uwfc_1                        |   File "/opt/specify7/specifyweb/specify/api.py", line 41, in toJson
uwfc_1                        |     return json.dumps(obj, cls=JsonEncoder)
uwfc_1                        |   File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
uwfc_1                        |     **kw).encode(obj)
uwfc_1                        |   File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
uwfc_1                        |     chunks = self.iterencode(o, _one_shot=True)
uwfc_1                        |   File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
uwfc_1                        |     return _iterencode(o, 0)
uwfc_1                        |   File "/opt/specify7/specifyweb/specify/api.py", line 37, in default
uwfc_1                        |     return obj.decode()
uwfc_1                        | UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 32527-32528: invalid continuation byte

[grantfitzsimmons]

Here is the exported file from Specify 6. It was created in 6, modified in Jaspersoft Studios and edited in 7 using formatting incompatible with Specify 6's report runner.

This is the JRXML of the report (in a styled doc, I apologize).

[maxpatiiuk]

From #1919 by @grantfitzsimmons:

In this case, I have an app resource that is throwing an error.

'utf-8' codec can't decode byte 0x88 in position 10: invalid start byte

I should be able to download a local copy and then delete the app resource from the UI. This error should (ideally) be friendly and allow those two options.

db: herb_rbge_0601 user: britchie branch: issue-1872 (unrelated presumably) /specify/viewsets/1/

[maxpatiiuk]

From @grantfitzsimmons:

UnicodeDecodeError at /api/specify/spappresourcedata/
'utf-8' codec can't decode byte 0x86 in position 10: invalid start byte

https://herbrbge6802-edge-2.test.specifysystems.org/specify/resources/view-set/3/

This is the view set named "Common".

Specify 7 Crash Report - 2022-12-16T13_56_39.401Z.txt

Same error on v7.7.5: https://herbrbge6802-v775.test.specifysystems.org/specify/viewsets/3/

On sandbox-rbge on the EU cloud, another 500 error with presumably a similar cause:

https://sandbox-rbge.specifycloud.org/specify/resources/app-resource/30/

[grantfitzsimmons]

RBGE is asking for a solution for this, even if it's a long-standing issue.

[maxpatiiuk]

@realVinayak do you know if your changes in https://github.com/specify/specify7/pull/4237 resolved this bug/affected this bug?

[emenslin]

Can recreate in edge (7.9.6) Tried to open the global resource Common in herbrbge Specify 7 Crash Report - 2024-07-26T15_56_43.955Z.txt