ReactiveHTML: Object of type Image is not JSON serializable

[MarcSkovMadsen]

panel: 0.12.4

My Pain

I am trying to create a ReactiveHTML ImageViewer for pillow Images. I want be able to provide a pillow Image and then transfer then image as a data url to the client side. I.e. the image should not be transfered.

But I get TypeError: Object of type Image is not JSON serializable

The problem here is that ReactiveHTML only supports JSON serializable parameters including parameters that it does not have to transfer.

import base64
import io

import panel as pn
import param
import PIL

class ImageViewer(pn.reactive.ReactiveHTML):
    image = param.ClassSelector(class_=PIL.Image.Image)
    data_url = param.String(default="test_string", constant=True)

    _template = """<img id="value" src="" style="height:100%;width:100%">${data_url}</img>"""

if __name__.startswith("bokeh"):
    image = PIL.Image.new('RGBA', size=(50, 50), color=(155, 0, 0))
    image.format = "PNG"
    ImageViewer(image=image).servable()

$ panel serve 'script.py' --autoreload --show
2021-11-15 20:29:35,627 Starting Bokeh server version 2.4.1 (running on Tornado 6.1)
2021-11-15 20:29:35,634 User authentication hooks NOT provided (default user enabled)
2021-11-15 20:29:35,636 Bokeh app running at: http://localhost:5006/script
2021-11-15 20:29:35,636 Starting Bokeh server with process id: 42852
2021-11-15 20:29:36,445 WebSocket connection opened
2021-11-15 20:29:36,445 ServerConnection created
2021-11-15 20:29:46,547 WebSocket connection closed: code=1001, reason=None
2021-11-15 20:29:46,847 WebSocket connection opened
2021-11-15 20:29:46,848 ServerConnection created
2021-11-15 20:29:46,852 error handling message
 message: Message 'PULL-DOC-REQ' content: {}
 error: TypeError('Object of type Image is not JSON serializable')
Traceback (most recent call last):
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\server\protocol_handler.py", line 97, in handle
    work = await handler(message, connection)
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\server\session.py", line 93, in _needs_document_lock_wrapper
    result = func(self, *args, **kwargs)
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\server\session.py", line 258, in _handle_pull
    return connection.protocol.create('PULL-DOC-REPLY', message.header['msgid'], self.document)
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\protocol\__init__.py", line 137, in create
    return self._messages[msgtype].create(*args, **kwargs)  # type: ignore [attr-defined]
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\protocol\messages\pull_doc_reply.py", line 85, in create  
    content = PullDoc(doc=document.to_json())
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\document\document.py", line 758, in to_json
    doc_json = self.to_json_string()
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\document\document.py", line 790, in to_json_string        
    return serialize_json(json, indent=indent)
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\core\json_encoder.py", line 171, in serialize_json        
    return json.dumps(obj, cls=BokehJSONEncoder, allow_nan=False, indent=indent, separators=separators, sort_keys=True, **kwargs)
  File "C:\python38\lib\json\__init__.py", line 234, in dumps
    return cls(
  File "C:\python38\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\python38\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\core\json_encoder.py", line 265, in default
    return self.transform_python_types(obj)
  File "c:\repos\private\panel-ai\.venv\lib\site-packages\bokeh\core\json_encoder.py", line 234, in transform_python_types
    return super().default(obj)
  File "C:\python38\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Image is not JSON serializable

[philippjfr]

Is this surprising? How would ReactiveHTML know how to serialize a PIL image?

[philippjfr]

Oh right, got you sorry. The solution here is to set a negative precedence for anything you don't want to serialize.

[MarcSkovMadsen]

Thanks. The solution to that bug would then be to document the solution it I guess.

[MarcSkovMadsen]

It would probably be even better to be able to specify in child_config or somewhere else that the parameter should not be serialized and transfered.

[jbednar]

And maybe make this a warning with a suggestion to set the precedence if you don't need the item serialized?

[philippjfr]

Not easy to do since it's non-trivial to determine what can and can't be serialized ahead of time.

[jbednar]

I guess it can't be a caught exception? (Since the exception presumably aborts the whole serialization step?)

[philippjfr]

The serialization happens in the whole application's context and at that point there is no easy way to map that back to the model being serialized or even the specific property/parameter being serialized.

[philippjfr]

Basically by the time you get to the serialization step it's already way too late.

[jbednar]

Param's JSON schema support has a mode that detects if serialization is potentially unsafe, but that would presumably be too trigger-happy here.

[philippjfr]

Param's JSON schema support has a mode that detects if serialization is potentially unsafe, but that would presumably be too trigger-happy here.

Significantly so, Param -> Bokeh Property conversion is significantly more powerful than what Param supports.