Open whitleykeith opened 3 months ago
To address the UnicodeDecodeError
you're encountering when querying binary, non-UTF-8 encodable data from a VARBINARY column in Trino using SQL Lab in Superset, consider implementing a custom JSON serializer that can handle binary data by encoding it in a format like base64. This will allow binary data to be serialized into a string format that JSON can handle without errors.
Here are the steps you can take:
Implement a Custom Serializer: Create a custom JSON serializer for handling binary data, ensuring it's encoded in a format such as base64 before serialization.
Modify JSON Serialization Calls: Update the json.dumps
calls in sqllab/api.py
to utilize this custom serializer, particularly where JSON serialization of execution results and data serialization occurs.
Adjust Data Decompression Logic: In utils/core.py
, ensure the decompression logic can handle binary data correctly, which may involve type checks before decompression and appropriate handling for binary data.
Thorough Testing: Test the querying of binary data from VARBINARY columns extensively to confirm the absence of the UnicodeDecodeError
.
Documentation: Update project documentation to reflect the handling of binary data in SQL Lab queries.
This approach should help you resolve the encoding issues you're facing with binary data in Superset's SQL Lab.
To continue the conversation, mention @dosu.
+1
We're running into this as well. It also affects the Table viz (and possibly others), not just SQL Lab.
@rusackas Could this have anything to do with the stringify
stuff you were trying to remove?
I don't think so, but it's possible. I was trying to allow UTF-8 strings through without them being escaped. It's possible this has to do with something being escaped incorrectly, which may or may not be the same chunk o' code.
it seems that simplejson will handle bytes itself, and will not use logic in 'default' function:
import simplejson
def convert_bytes(val):
if isinstance(val, bytes):
return 'converted-' + repr(val)
return val
print(simplejson.__version__)
# 3.17.3
print(simplejson.dumps({'a': b'b'}, default=convert_bytes))
# {"a": "b"}
print(simplejson.dumps({'a': b'\x00\x85'}, default=convert_bytes))
# raises UnicodeDecodeError
so bytes should be handled before they goes into the dumps
function.
Bug description
See https://github.com/apache/superset/issues/28001 for context
I attempted to validate the fix from https://github.com/apache/superset/pull/28266 but I'm still getting the same error as described in the issue.
How to reproduce the bug
Screenshots/recordings
No response
Superset version
467e612-dev
Python version
3.10
Node version
I don't know
Browser
Chrome
Additional context
Stack trace:
Also, sometimes the stack trace and error message can be slightly different. I can see this stack trace too:
Checklist