Adding Class Representer to YAML Results in Nonsense Error

[JacobLChrzanowski]

Hello,

I am trying to load and store a config from a yaml, and I want to have a custom class able to be dumped and loaded from yaml. I have this working outside of HASS/PyScript, but it totally bites the dust in PyScript.

Versions

Home Assistant Core: 2023.8.4 Home Assistant OS: 10.5 PyScript version: 1.5.0 HASS PyYAML library: 6.0.1 Debian PyYAML library: 6.0.1, and also worked on 5.3.1

(venv) jacobc@webprime:~/python_venv$ pip list | grep -i yaml
PyYAML        6.0.1
jacobc@webprime:~/python_venv$ deactivate
jacobc@webprime:~/python_venv$ pip list | grep -i yaml
PyYAML              5.3.1

Here we go, the MVP:

import yaml
from enum import Enum, Flag, auto
Action = Flag('Action', ['Press', 'Hold', 'TerminateHold', 'Indeterminate', 'Up', 'Down', 'Left', 'Right', 'Center'])

def Action_representer(dumper, data: Action):
    """
    Custom representer for an enum.Flag subclass 'Action' to enable customized YAML dumping.
    The function works by:
      converting the binary representation of the Action value into a set of power-of-two components,
      joining their corresponding Action names with '|',
      and then representing the result as a scalar in the YAML format.
    For instance:
        Action.Hold -> "!Action 'Hold'"
        Action.Hold | Action.Center -> "!Action 'Hold|Center'"
    Args:
      dumper: YAML dumper instance.
      data (Action): The Action object to be represented in the YAML format.
    Returns:
      Represents the Action object as a scalar in the custom YAML format.
    """
    binary_str = bin(data.value)[2:][::-1]
    binary_pieces = [2**(i+0) for i, bit in enumerate(binary_str) if bit == '1']
    repr_str = '|'.join([Action(x).name for x in binary_pieces])
    return dumper.represent_scalar(u'!Action', repr_str)
yaml.add_representer(Action, Action_representer)

print(yaml.dump(Action.Hold))

Debian Host:

This outputs as expected in an ipython shell,

In [4]: print(yaml.dump(Action.Hold))
!Action 'Hold'

Inside of PyScript:

Before running yaml.add_representer(Action, Action_representer)

log.info(yaml.dump({'hello': Action.Hold}))
->
2023-10-23 22:16:21.202 INFO (MainThread) [custom_components.pyscript.file.example.async_foo] hello: !!python/object/apply:builtins.getattr
- !!python/name:custom_components.pyscript.eval.Action ''
- Hold

After running yaml.add_representer(Action, Action_representer)

log.info(yaml.dump({'hello': Action.Hold}))
->
2023-10-23 22:16:41.349 ERROR (MainThread) [custom_components.pyscript.file.example.async_foo] Exception in <file.example.async_foo> line 196:
        log.info(yaml.dump({'hello': Action.Hold}))
                            ^
EmitterError: expected NodeEvent, but got MappingEndEvent()

No clue where to even start here, there isn't much on Google that is very helpful. Anyone have thoughts? Ideas?

All the best, Jacob C.

[JacobLChrzanowski]

Disappointingly, YAML -> Python Object fails too.

MVP:

from enum import Enum, Flag, auto
import yaml
Action = Flag('Action', ['Press', 'Hold', 'TerminateHold', 'Indeterminate', 'Up', 'Down', 'Left', 'Right', 'Center'])

def Action_constructor(loader, node):
    value = loader.construct_scalar(node)
    values = value.split('|')
    action_obj = Action(0)
    for action in values:
        action_obj |= Action[action] 
    return action_obj
yaml.add_constructor(u'!Action', Action_constructor)

On a Debian host:

In [28]: yaml.load("!Action 'Hold|TerminateHold'\n", Loader=yaml.FullLoader)
['Hold', 'TerminateHold']
Out[28]: <Action.TerminateHold|Hold: 6>

In PyScript, it does not error, but it does not return a useable object either:

log.info(yaml.load("!Action 'Hold'", Loader=yaml.Loader))
->
2023-10-23 22:38:36.544 INFO (MainThread) [custom_components.pyscript.file.example.async_foo] <coroutine object EvalFuncVar.__call__ at 0x7f7477c220>

[JacobLChrzanowski]

Ok so, shoving things into a @pyscript_compile tag makes them work as expected but is this a 'solution'? PyScript docs state this is not totally expected to stick around. 'This is an experimental feature and might change in the future.' unless I misunderstood what this means.

See it in action here: https://github.com/JacobLChrzanowski/HomeAssistant/blob/cbce3b1d5703c750a2df7261a7eb427a22f28b7d/CONFIG/pyscript/example.py#L170

@pyscript_compile
def test_func():
    def Action_representer(dumper, data: Action | ActionMode):
        """
        Custom representer for an enum.Flag subclass 'Action' to enable customized YAML dumping.
        The function works by:
        converting the binary representation of the Action value into a set of power-of-two components,
        joining their corresponding Action names with '|',
        and then representing the result as a scalar in the YAML format
        Args:
        dumper: YAML dumper instance.
        data (Action): The Action object to be represented in the YAML format.
        Returns:
        Represents the Action object as a scalar in the custom YAML format.
        """
        class_name = f"!{data.__class__.__name__}"
        binary_str = bin(data.value)[2:][::-1]
        binary_pieces = [2**(i+0) for i, bit in enumerate(binary_str) if bit == '1']
        repr_str = '|'.join([Action(x).name for x in binary_pieces])
        return dumper.represent_scalar(class_name, repr_str)
    yaml.add_representer(Action, Action_representer)
    yaml.add_representer(ActionMode, Action_representer)

    def Action_constructor(loader, node):
        value = loader.construct_scalar(node)
        values = value.split('|')
        action_obj = Action(0)
        for action in values:
            action_obj |= Action[action] 
        return action_obj
    yaml.add_constructor(u'!Action', Action_constructor)

[craigbarratt]

One reason (and maybe the only reason) why the yaml package won't work with pyscript is that the callback functions you register via calls like yaml.add_constructor() or yaml.add_representer()need to be regular functions. However, all pyscript functions are async, so they won't work as callbacks.

The solution as you discovered is @pyscript_compile, which turns the function into a regular (complied) Python function. So your solution is a good one. It's possible that it will also work correctly if you just apply the @pyscript_compile to each of the two inner callback functions, instead of the whole function.

An alternative is to put the code into a module that you can import, since that will be treated as native compiled Python code. See the docs.

I'll remove that comment in the docs, since it's no longer true. When I first added @pyscript_compile I wasn't sure it was a useful or appropriate feature.

[JacobLChrzanowski]

Sorry, I read your answer and implemented fixes over a month ago, totally forgot to reply! Thanks @craigbarratt !