New UnicodeEncodeError: 'utf-8' for tempalte rendering

The problem

The template parser doesn't recognize anymore the more-than-one UTF-8 sequences for HTML codes. The core parser logs the following error: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 155-156: surrogates not allowed

Examples: 🔋 using UTF-8: \uD83D\uDD0B 🐌 using UTF-8: \uD83D\uDC0C 🌡️ using UTF-8: \uD83C\uDF21\uFE0F

The error applies for:

Template parser
Wear OS
Companion app plugins

What version of Home Assistant Core has the issue?

System Information version | core-2024.1.1 -- | -- installation_type | Home Assistant OS dev | false hassio | true docker | true user | root virtualenv | false python_version | 3.11.6 os_name | Linux os_version | 6.1.70-haos arch | x86_64 timezone | Europe/Lisbon config_dir | /config
Home Assistant Community Store
GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 4982 Installed Version | 1.33.0 Stage | running Available Repositories | 1377 Downloaded Repositories | 45

Home Assistant Cloud
logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok

Home Assistant Supervisor
host_os | Home Assistant OS 11.3 -- | -- update_channel | stable supervisor_version | supervisor-2023.12.0 agent_version | 1.6.0 docker_version | 24.0.7 disk_total | 30.8 GB disk_used | 14.3 GB healthy | true supported | true board | ova supervisor_api | ok version_api | ok installed_addons | Grafana (9.1.1), Let's Encrypt (5.0.9), File editor (5.7.0), Duck DNS (1.15.0), Studio Code Server (5.14.2), Samba share (12.2.0), Home Assistant Google Drive Backup (0.112.1), Check Home Assistant configuration (3.11.0), SQLite Web (4.0.0), InfluxDB (4.8.0), Node-RED (16.0.2), Mosquitto broker (6.4.0)

Dashboards
dashboards | 3 -- | -- resources | 29 views | 8 mode | storage

Recorder
oldest_recorder_run | January 4, 2024 at 10:34 AM -- | -- current_recorder_run | January 5, 2024 at 10:06 PM estimated_db_size | 571.03 MiB database_engine | sqlite database_version | 3.41.2

Sonoff
version | 3.5.4 (a4a8c5f) -- | -- cloud_online | 12 / 13 local_online | 12 / 12

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

Before this problem it was possible these template layout examples...

Example YAML snippet

Try the templating at the /developer-tools/template
with and without multi-escape sequences, like:

{{"\u231B"}} 
... works ok

and

{{"\u231B"}}
{{"\uD83C\uDF21\uFE0F"}}
... fails!

Anything in the logs that might be useful for us?

2024-01-05 11:55:03.886 ERROR (MainThread) [homeassistant.components.websocket_api.messages] Unable to serialize to JSON. Bad data found at $.event.result=\ud83d\udc0c(<class 'str'>
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 155-156: surrogates not allowed
Call stack:
  File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
Message: "Unable to serialize to JSON. Bad data found at $.event.result=\ud83d\udc0c(<class 'str'>"

Additional information

No response

home-assistant / core