legrego / homeassistant-elasticsearch

Publish Home-Assistant events to Elasticsearch
https://legrego.github.io/homeassistant-elasticsearch/
MIT License
149 stars 37 forks source link

[2.0] Unable to serialize JSON Data #346

Open strawgate opened 3 months ago

strawgate commented 3 months ago

Error in publishing loop:

Unknown transport error connecting to Elasticsearch: Unable to serialize to JSON: {'@timestamp': '2024-08-23T15:55:19.198956+00:00', 'event.action': 'Polling', 'event.kind': 'event', 'event.type': 'info', 'hass.entity.attributes': {'access_token': 'token', 'rooms': '{"1": {"x0": -7950, "y0": -1200, "x1": -1950, "y1": 1150, "room_id": 1, "name": "Dining Hall", "type": 5, "index": 0, "icon": "mdi:room-service-outline", "color_index": 2, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": -3600, "y": 50}, "2": {"x0": -4150, "y0": -2800, "x1": -2000, "y1": -1150, "room_id": 2, "name": "Bathroom", "type": 6, "index": 0, "icon": "mdi:toilet", "color_index": 3, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": -3100, "y": -1700}, "3": {"x0": -2050, "y0": -2400, "x1": 2900, "y1": 1050, "room_id": 3, "name": "Living Room", "type": 1, "index": 0, "icon": "mdi:sofa-outline", "color_index": 0, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": 450, "y": -600}, "4": {"x0": -1900, "y0": -5900, "x1": 3100, "y1": -2050, "room_id": 4, "name": "Kitchen", "type": 4, "index": 0, "icon": "mdi:chef-hat", "color_index": 1, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": 700, "y": -3250}, "5": {"x0": -5150, "y0": -6200, "x1": -1600, "y1": -3750, "room_id": 5, "name": "Entry", "type": 0, "index": 0, "icon": "mdi:home-outline", "color_index": 0, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": -3400, "y": -4950}}', 'map_id': 80, 'map_name': 'Map 1', 'rotation': 0, 'pathways': [], 'carpets': [[-775, -1975, 2325, -1975, 2325, 375, -775, 375]], 'ignored_carpets': [], 'detected_carpets': [[-3925, -2025, -2925, -2025, -2925, -1325, -3925, -1325], [-775, -2025, 2325, -2025, 2325, 425, -775, 425], [2175, -4275, 3025, -4275, 3025, -3025, 2175, -3025], [-4775, -6125, -4125, -6125, -4125, -4925, -4775, -4925], [1925, -2075, 2075, -2075, 2075, -1975, 1925, -1975]], 'is_empty': False, 'map_index': 1, 'furnitures': [(-3925, -5625), (1125, 825)], 'recovery_map_list': '[{"date": "2024-02-18 21:29", "map_type": "Edited", "object_name": "ali_dreame/KC415732/722917896/101"}, {"date": "2024-02-17 21:17", "map_type": "Original", "object_name": "ali_dreame/KC415732/722917896/100"}]', 'calibration_points': None, 'selected': True, 'recovery_map_picture': '{"02/18/24 21:29:01: Map2 (Edited)": "/api/camera_recovery_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&index=2&v=1708313341", "02/17/24 21:17:15: Map1 (Original)": "/api/camera_recovery_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&index=1&v=1708226235"}', 'recovery_map_file': '{"02/18/24 21:29:01: Map2 (Edited)": "/api/camera_recovery_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&index=2&v=1708313341&file=1", "02/17/24 21:17:15: Map1 (Original)": "/api/camera_recovery_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&index=1&v=1708226235&file=1"}', 'wifi_map_picture': '/api/camera_wifi_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&v=1724428455', 'supported_features': <CameraEntityFeature: 0>}, 'hass.entity.domain': 'camera', 'hass.entity.id': 'camera.l10s_ultra_map_1', 'hass.entity.value': 'unavailable', 'hass.entity.valueas': {'string': 'unavailable'}, 'hass.entity.object.id': 'l10s_ultra_map_1', 'datastream.type': 'metrics', 'datastream.dataset': 'homeassistant.camera', 'datastream.namespace': 'default', 'hass.entity.area.id': 'living_room', 'hass.entity.area.name': 'Living Room', 'hass.entity.friendly_name': 'L10s Ultra Saved Map 1', 'hass.entity.platform': 'dreame_vacuum', 'hass.entity.device.name': 'L10s Ultra', 'hass.entity.device.area.id': 'living_room', 'hass.entity.device.area.name': 'Living Room', 'agent.version': '2024.7.4', 'host.architecture': 'aarch64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant'} (type: dict)

Also could not seem to exclude this with a label so will look at that too

legrego commented 2 months ago

My guess is that we failed to serialize the custom CameraEntityFeature value:

'supported_features': <CameraEntityFeature: 0>

I would have expected this to get caught by filter_attribute, but perhaps there's something attempting serialization before the filter step?

legrego commented 2 months ago

^ I'm not entirely convinced of this anymore. I believe there's something incompatible within the attribute list provided by this integration (https://github.com/Tasshack/dreame-vacuum) that our filtering doesn't account for, but I haven't figured out what that is, yet.

Here's a formatted version of the failed document from the OP:

{
    "@timestamp": "2024-08-23T15:55:19.198956+00:00",
    "event.action": "Polling",
    "event.kind": "event",
    "event.type": "info",
    "hass.entity.attributes": {
        "access_token": "token",
        "rooms": '{"1": {"x0": -7950, "y0": -1200, "x1": -1950, "y1": 1150, "room_id": 1, "name": "Dining Hall", "type": 5, "index": 0, "icon": "mdi:room-service-outline", "color_index": 2, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": -3600, "y": 50}, "2": {"x0": -4150, "y0": -2800, "x1": -2000, "y1": -1150, "room_id": 2, "name": "Bathroom", "type": 6, "index": 0, "icon": "mdi:toilet", "color_index": 3, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": -3100, "y": -1700}, "3": {"x0": -2050, "y0": -2400, "x1": 2900, "y1": 1050, "room_id": 3, "name": "Living Room", "type": 1, "index": 0, "icon": "mdi:sofa-outline", "color_index": 0, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": 450, "y": -600}, "4": {"x0": -1900, "y0": -5900, "x1": 3100, "y1": -2050, "room_id": 4, "name": "Kitchen", "type": 4, "index": 0, "icon": "mdi:chef-hat", "color_index": 1, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": 700, "y": -3250}, "5": {"x0": -5150, "y0": -6200, "x1": -1600, "y1": -3750, "room_id": 5, "name": "Entry", "type": 0, "index": 0, "icon": "mdi:home-outline", "color_index": 0, "floor_material": 1, "floor_material_direction": "Horizontal", "visibility": "Visible", "x": -3400, "y": -4950}}',
        "map_id": 80,
        "map_name": "Map 1",
        "rotation": 0,
        "pathways": [],
        "carpets": [[-775, -1975, 2325, -1975, 2325, 375, -775, 375]],
        "ignored_carpets": [],
        "detected_carpets": [
            [-3925, -2025, -2925, -2025, -2925, -1325, -3925, -1325],
            [-775, -2025, 2325, -2025, 2325, 425, -775, 425],
            [2175, -4275, 3025, -4275, 3025, -3025, 2175, -3025],
            [-4775, -6125, -4125, -6125, -4125, -4925, -4775, -4925],
            [1925, -2075, 2075, -2075, 2075, -1975, 1925, -1975],
        ],
        "is_empty": False,
        "map_index": 1,
        "furnitures": [(-3925, -5625), (1125, 825)],
        "recovery_map_list": '[{"date": "2024-02-18 21:29", "map_type": "Edited", "object_name": "ali_dreame/KC415732/722917896/101"}, {"date": "2024-02-17 21:17", "map_type": "Original", "object_name": "ali_dreame/KC415732/722917896/100"}]',
        "calibration_points": None,
        "selected": True,
        "recovery_map_picture": '{"02/18/24 21:29:01: Map2 (Edited)": "/api/camera_recovery_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&index=2&v=1708313341", "02/17/24 21:17:15: Map1 (Original)": "/api/camera_recovery_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&index=1&v=1708226235"}',
        "recovery_map_file": '{"02/18/24 21:29:01: Map2 (Edited)": "/api/camera_recovery_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&index=2&v=1708313341&file=1", "02/17/24 21:17:15: Map1 (Original)": "/api/camera_recovery_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&index=1&v=1708226235&file=1"}',
        "wifi_map_picture": "/api/camera_wifi_map_proxy/camera.l10s_ultra_map_1?token=21cec19731f8b180cd36fa40a5f32b8960ed2d871ae8ec7a9ad47c49cd53ffac&v=1724428455",
        "supported_features": <CameraEntityFeature: 0>,
    },
    "hass.entity.domain": "camera",
    "hass.entity.id": "camera.l10s_ultra_map_1",
    "hass.entity.value": "unavailable",
    "hass.entity.valueas": {"string": "unavailable"},
    "hass.entity.object.id": "l10s_ultra_map_1",
    "datastream.type": "metrics",
    "datastream.dataset": "homeassistant.camera",
    "datastream.namespace": "default",
    "hass.entity.area.id": "living_room",
    "hass.entity.area.name": "Living Room",
    "hass.entity.friendly_name": "L10s Ultra Saved Map 1",
    "hass.entity.platform": "dreame_vacuum",
    "hass.entity.device.name": "L10s Ultra",
    "hass.entity.device.area.id": "living_room",
    "hass.entity.device.area.name": "Living Room",
    "agent.version": "2024.7.4",
    "host.architecture": "aarch64",
    "host.os.name": "Linux",
    "host.hostname": "homeassistant",
}
strawgate commented 2 months ago

I had this issue previously with the old version of the integration too.

I'll try to find some time in the next couple days to look closer.

Also looks like I may have not done the best job scrubbing the token...