search

legrego / homeassistant-elasticsearch

Publish Home-Assistant events to Elasticsearch
https://legrego.github.io/homeassistant-elasticsearch/
MIT License
148 stars 37 forks source link

Error publishing documents to Elasticsearch #63

Closed johntdyer closed 4 years ago

johntdyer commented 5 years ago

Environment Home-Assistant version: 0.92.4 Elasticsearch version: 6.7

Relevant configuration.yml settings:

# Do not include your Elasticsearch URL, credentials, or any other sensitive information
elastic:
  url: http://localhost:9200
  ...

Describe the bug

getting the following in my logs ( truncated ) 

2019-04-19 07:49:56 INFO (MainThread) [elasticsearch] POST http://localhost:9200/_bulk [status:200 request:0.127s]
2019-04-19 07:49:56 ERROR (MainThread) [custom_components.elastic] Error publishing documents to Elasticsearch: ('2 document(s) failed to index.', [{'index': {'_index': 'hass-events-v2-000001', '_type': 'doc', '_id': 'lQ9uNWoBlgpy3TEwgkab', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'mapper [hass.attributes.py_version] of different type, current_type [text], merged_type [long]'}, 'data': {'hass.domain': 'alarm_control_panel', 'hass.object_id': 'house', 'hass.entity_id': 'alarm_control_panel.house', 'hass.attributes': {'code_format': '.+', 'changed_by': '', 'immediate': set(), 'delayed': set(), 'ignored': {'binary_sensor.home_sensor', 'binary_sensor.away_delayed_sensor', 'binary_sensor.home_delayed_sensor', 'binary_sensor.perimeter_delayed_sensor', 'binary_sensor.perimeter_sensor', 'switch.skylight', 'binary_sensor.away_sensor'}, 'allsensors': {'binary_sensor.home_sensor', 'binary_sensor.away_delayed_sensor', 'binary_sensor.home_delayed_sensor', 'binary_sensor.perimeter_delayed_sensor', 'binary_sensor.perimeter_sensor', 'switch.skylight', 'binary_sensor.away_sensor'}, 'code_to_arm': False, 'panel_locked': False, 'passcode_attempts': 2, 'passcode_attempts_timeout': 10, 'changedbyuser': None, 'panic_mode': 'deactivated', 'arm_state': 'disarmed', 'enable_perimeter_mode': True, 'enable_persistence': False, 'enable_log': True, 'log_size': 10, 'supported_statuses_on': ['on', 'true', 'unlocked', 'open', 'detected', 'motion', 'motion_detected', 'motion detected'], 'supported_statuses_off': ['off', 'false', 'locked', 'closed', 'undetected', 'no_motion', 'standby'], 'updateUI': False, 'admin_password': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'bwalarm_version': '1.1.3', 'py_version': sys.version_info(major=3, minor=6, micro=7, releaselevel='final', serial=0), 'users': [OrderedDict([('id', 'xxxxxxxx'), ('name', 'test'), ('enabled', False), ('code', '****'), ('picture', '/local/images/ha.png'), ('disable_animations', False)]), OrderedDict([('id', 'xxxxxxxx'), ('name', 'bart'), ('enabled', True), ('code', '****'), ('picture', '/local/images/hal.png'), ('disable_animations', False)]), OrderedDict([('id', 'xxxxxxxxxxxx'), ('name', 'Legacy API password user'), ('enabled', False), ('code', '****'), ('picture', '/local/images/ha.png'), ('disable_animations', False)]), OrderedDict([('id', 'xxxxxxxxxxx'), ('name', 'Legacy API password user'), ('enabled', False), ('code', '****'), ('picture', '/local/images/ha.png'), ('disable_animations', False)])], 'panel': OrderedDict([('camera_update_interval', ''), ('cameras', []), ('enable_camera_panel', 'False'), ('enable_clock', 'True'), ('enable_clock_12hr', 'True'), ('enable_custom_panel', 'False'), ('enable_floorplan_panel', 'False'), ('enable_sensors_panel', 'True'), ('enable_serif_font', 'True'), ('enable_weather', 'True'), ('hide_passcode', 'True'), ('panel_title', 'Surname Residence'), ('shadow_effect', 'True'), ('enable_fahrenheit', 'False')]), 'themes': [OrderedDict([('name', 'aaa'), ('warning_color', '#995BFF'), ('pending_color', '#FF2943'), ('disarmed_color', '#FF22E6'), ('triggered_color', '#FF0000'), ('armed_home_color', '#C1B1FF'), ('armed_away_color', '#FF8686'), ('armed_perimeter_color', '#DAFF9E'), ('active', 'False'), ('action_button_border_color', '#3ED5FF')])], 'logs': [], 'mqtt': {'enable_mqtt': False, 'payload_arm_night': 'ARM_NIGHT', 'state_topic': 'home/alarm', 'payload_disarm': 'DISARM', 'payload_arm_home': 'ARM_HOME', 'qos': 0, 'override_code': False, 'command_topic': 'home/alarm/set', 'pending_on_warning': False, 'payload_arm_away': 'ARM_AWAY'}, 'states': OrderedDict([('armed_away', OrderedDict([('immediate', ['switch.skylight', 'binary_sensor.away_sensor']), ('delayed', ['binary_sensor.away_delayed_sensor']), ('override', []), ('pending_time', 2), ('warning_time', 2), ('trigger_time', 5)])), ('armed_home', OrderedDict([('immediate', ['binary_sensor.home_sensor']), ('delayed', ['binary_sensor.home_delayed_sensor']), ('override', []), ('pending_time', 2), ('warning_time', 2), ('trigger_time', 5)])), ('armed_perimeter', OrderedDict([('immediate', ['binary_sensor.perimeter_sensor']), ('delayed', ['binary_sensor.perimeter_delayed_sensor']), ('override', []), ('pending_time', 0), ('warning_time', 2), ('trigger_time', 600)]))]), 'friendly_name': 'House'}, 'hass.value': 'disarmed', '@timestamp': datetime.datetime(2019, 4, 19, 7, 49, 55, 950180), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.91.4', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 34.xxxxx, 'lon': -84.xxxxxx}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'ubuntu', 'tags': ['hass']}}}, {'index': {'_index': 'hass-events-v2-000001', '_type': 'doc', '_id': 'pQ9uNWoBlgpy3TEwgkab', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.attributes.media_content_id] of type [long] in document with id 'pQ9uNWoBlgpy3TEwgkab'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "spotify:track:xxxxxxx"'}}, 'data': {'hass.domain': 'media_player', 'hass.object_id': 'spotify', 'hass.entity_id': 'media_player.spotify', 'hass.attributes': {'media_content_id': 'spotify:track:xxxxxxxxxxx', 'media_content_type': 'music', 'media_title': 'I Was Jack (You Were Diane)', 'media_artist': 'Jake Owen', 'media_album_name': 'Greetings From...Jake', 'source': 'Office', 'source_list': ['JOHNDYE-M-D3KG', '50" TCL Roku TV', 'Roku TV'], 'shuffle': False, 'friendly_name': 'Spotify', 'icon': 'mdi:spotify', 'entity_picture': '/api/media_player_proxy/media_player.spotify?token=xxxxxxx121ee&cache=xxxxx', 'supported_features': 51765}, 'hass.value': 'paused', '@timestamp': datetime.datetime(2019, 4, 19, 7, 49, 55, 950180), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.91.4', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 34.xxxxxx, 'lon': -84.xxxx}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'ubuntu', 'tags': ['hass']}}}])
Traceback (most recent call last):
  File "/home/homeassistant/.homeassistant/custom_components/elastic/__init__.py", line 324, in do_publish
    bulk_response = bulk(self._gateway.get_client(), actions)
  File "/srv/homeassistant/lib/python3.6/site-packages/elasticsearch/helpers/__init__.py", line 257, in bulk
    for ok, item in streaming_bulk(client, actions, *args, **kwargs):
  File "/srv/homeassistant/lib/python3.6/site-packages/elasticsearch/helpers/__init__.py", line 192, in streaming_bulk
    raise_on_error, *args, **kwargs)
  File "/srv/homeassistant/lib/python3.6/site-packages/elasticsearch/helpers/__init__.py", line 137, in _process_bulk_chunk
    raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)
legrego commented 5 years ago

Thanks for the report @johntdyer.

If possible, can you query ES and see what HASS entities have been published with a hass.attributes.py_version field? I'm trying to figure out if there are multiple entities using that specific field, but reporting the value differently.

Currently, hass.attributes is a dynamic bucket of properties, and I've been trusting HASS to not have one entity clobber the properties of another entity.

johntdyer commented 5 years ago

@legrego - I'm happy to perform any queries you want, just not sure how you'd like me to prove or disprove your theory?

johntdyer commented 5 years ago

So FWIW this is the error I am getting now

2019-07-19 13:48:33 ERROR (MainThread) [custom_components.elastic] Error publishing documents to Elasticsearch: ('89 document(s) failed to index.',

and I extracted and redacted a sample of events to hopefully help diagnose :

this is an example of the events Hass is sending

[

  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '9o1ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '9o1ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "idle"'}}, 'data': {'hass.domain': 'camera', 'hass.object_id': 'ffc', 'hass.entity_id': 'camera.ffc', 'hass.attributes': {'access_token': 'xxxxxxxxxx', 'friendly_name': 'FFC', 'entity_picture': '/api/camera_proxy/camera.ffc?token=xxxxxxxxxxxxx', 'supported_features': 0}, 'hass.value': 'idle', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 9, 5498, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},

  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '941ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '941ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "idle"'}}, 'data': {'hass.domain': 'camera', 'hass.object_id': 'unifi_driveway', 'hass.entity_id': 'camera.unifi_driveway', 'hass.attributes': {'access_token': 'xxxxxxxxxx', 'model_name': 'UVC G3', 'brand': 'Ubiquiti', 'motion_detection': True, 'friendly_name': 'UniFi Driveway', 'entity_picture': '/api/camera_proxy/camera.unifi_driveway?token=xxxxxxxxxxxxx', 'supported_features': 0}, 'hass.value': 'idle', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 9, 181421, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '-I1ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '-I1ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "idle"'}}, 'data': {'hass.domain': 'camera', 'hass.object_id': 'unifi_living_room', 'hass.entity_id': 'camera.unifi_living_room', 'hass.attributes': {'access_token': 'xxxxxxxxxx', 'model_name': 'UVC G3 Flex', 'brand': 'Ubiquiti', 'motion_detection': True, 'friendly_name': 'UniFi Living Room', 'entity_picture': '/api/camera_proxy/camera.unifi_living_room?token=xxxxxxxxxxxxx', 'supported_features': 0}, 'hass.value': 'idle', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 9, 312246, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '-Y1ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '-Y1ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "idle"'}}, 'data': {'hass.domain': 'camera', 'hass.object_id': 'unifi_front_door', 'hass.entity_id': 'camera.unifi_front_door', 'hass.attributes': {'access_token': 'xxxxxxxxxx', 'model_name': 'UVC G3', 'brand': 'Ubiquiti', 'motion_detection': True, 'friendly_name': 'UniFi Front Door', 'entity_picture': '/api/camera_proxy/camera.unifi_front_door?token=xxxxxxxxxxxxx', 'supported_features': 0}, 'hass.value': 'idle', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 9, 422018, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '-o1ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '-o1ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "idle"'}}, 'data': {'hass.domain': 'camera', 'hass.object_id': 'unifi_upstairs', 'hass.entity_id': 'camera.unifi_upstairs', 'hass.attributes': {'access_token': 'xxxxxxxxxx', 'model_name': 'UVC G3 Dome', 'brand': 'Ubiquiti', 'motion_detection': True, 'friendly_name': 'UniFi Upstairs', 'entity_picture': '/api/camera_proxy/camera.unifi_upstairs?token=xxxxxxxxxxxxx', 'supported_features': 0}, 'hass.value': 'idle', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 9, 532483, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '-41ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '-41ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "idle"'}}, 'data': {'hass.domain': 'camera', 'hass.object_id': 'unifi_backyard', 'hass.entity_id': 'camera.unifi_backyard', 'hass.attributes': {'access_token': 'xxxxxxxxxx', 'model_name': 'UVC G3 Flex', 'brand': 'Ubiquiti', 'motion_detection': True, 'friendly_name': 'UniFi Backyard', 'entity_picture': '/api/camera_proxy/camera.unifi_backyard?token=xxxxxxxxxxxxx', 'supported_features': 0}, 'hass.value': 'idle', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 9, 692237, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '_I1ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '_I1ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "idle"'}}, 'data': {'hass.domain': 'camera', 'hass.object_id': 'unifi_basement', 'hass.entity_id': 'camera.unifi_basement', 'hass.attributes': {'access_token': 'xxxxxxxxxx', 'model_name': 'UVC G3 Flex', 'brand': 'Ubiquiti', 'motion_detection': True, 'friendly_name': 'UniFi Basement', 'entity_picture': '/api/camera_proxy/camera.unifi_basement?token=xxxxxxxxxxxxx', 'supported_features': 0}, 'hass.value': 'idle', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 9, 880835, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '_Y1ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '_Y1ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "idle"'}}, 'data': {'hass.domain': 'camera', 'hass.object_id': 'unifi_server_room', 'hass.entity_id': 'camera.unifi_server_room', 'hass.attributes': {'access_token': 'xxxxxxxxxx', 'model_name': 'UVC G3 Flex', 'brand': 'Ubiquiti', 'motion_detection': True, 'friendly_name': 'UniFi Server Room', 'entity_picture': '/api/camera_proxy/camera.unifi_server_room?token=xxxxxxxxxxxxx', 'supported_features': 0}, 'hass.value': 'idle', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 10, 79147, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '_o1ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '_o1ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "idle"'}}, 'data': {'hass.domain': 'camera', 'hass.object_id': 'unifi_garage', 'hass.entity_id': 'camera.unifi_garage', 'hass.attributes': {'access_token': 'xxxxxxxxxx', 'model_name': 'UVC G3 Flex', 'brand': 'Ubiquiti', 'motion_detection': True, 'friendly_name': 'UniFi Garage', 'entity_picture': '/api/camera_proxy/camera.unifi_garage?token=xxxxxxxxxxxxx', 'supported_features': 0}, 'hass.value': 'idle', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 10, 251432, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': '_41ZC2wBLpP0xwShibSp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id '_41ZC2wBLpP0xwShibSp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "initializing"'}}, 'data': {'hass.domain': 'zwave', 'hass.object_id': 'ge_14291_in_wall_smart_switch_12', 'hass.entity_id': 'zwave.ge_14291_in_wall_smart_switch_12', 'hass.attributes': {'node_id': 31, 'node_name': 'Jasco Products 14291 In-Wall Smart Switch', 'manufacturer_name': 'Jasco Products', 'product_name': '14291 In-Wall Smart Switch', 'query_stage': 'CacheLoad', 'is_awake': True, 'is_ready': False, 'is_failed': False, 'is_info_received': True, 'max_baud_rate': 40000, 'is_zwave_plus': True, 'capabilities': {'zwave_plus', 'listening', 'beaming', 'routing'}, 'sentCnt': 3, 'sentFailed': 0, 'retries': 0, 'receivedCnt': 0, 'receivedDups': 0, 'receivedUnsolicited': 0, 'sentTS': '2019-07-19 13:48:00:666 ', 'receivedTS': '2019-07-19 13:43:54:928 ', 'lastRequestRTT': 47, 'averageRequestRTT': 47, 'lastResponseRTT': 0, 'averageResponseRTT': 0, 'friendly_name': 'Jasco Products 14291 In-Wall Smart Switch', 'hidden': True}, 'hass.value': 'initializing', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 10, 771935, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'AI1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'AI1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "ready"'}}, 'data': {'hass.domain': 'zwave', 'hass.object_id': 'aeotec_zw090_z_stick_gen5_us', 'hass.entity_id': 'zwave.aeotec_zw090_z_stick_gen5_us', 'hass.attributes': {'node_id': 1, 'node_name': 'AEON Labs ZW090 Z-Stick Gen5 US', 'manufacturer_name': 'AEON Labs', 'product_name': 'ZW090 Z-Stick Gen5 US', 'query_stage': 'Complete', 'is_awake': True, 'is_ready': True, 'is_failed': False, 'is_info_received': True, 'max_baud_rate': 40000, 'is_zwave_plus': False, 'capabilities': {'primaryController', 'listening', 'beaming'}, 'neighbors': {34, 19}, 'sentCnt': 1, 'sentFailed': 0, 'retries': 0, 'receivedCnt': 0, 'receivedDups': 0, 'receivedUnsolicited': 0, 'sentTS': '2019-07-19 13:44:03:748 ', 'receivedTS': '2019-07-19 13:43:54:923 ', 'lastRequestRTT': 0, 'averageRequestRTT': 0, 'lastResponseRTT': 0, 'averageResponseRTT': 0, 'friendly_name': 'AEON Labs ZW090 Z-Stick Gen5 US', 'hidden': True}, 'hass.value': 'ready', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 11, 301929, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'AY1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'AY1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "55d 7hr 45min"'}}, 'data': {'hass.domain': 'sensor', 'hass.object_id': 'server_room_gateway_uptime', 'hass.entity_id': 'sensor.server_room_gateway_uptime', 'hass.attributes': {'friendly_name': 'Server Room Gateway Uptime'}, 'hass.value': '55d 7hr 45min', '@timestamp': datetime.datetime(2019, 7, 19, 17, 48, 21, 110931, tzinfo=<UTC>), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'FI1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'FI1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "zoning"'}}, 'data': {'hass.domain': 'zone', 'hass.object_id': 'home', 'hass.entity_id': 'zone.home', 'hass.attributes': {'hidden': True, 'latitude': 34.13596802526734, 'longitude': -84.3395096833913, 'radius': 200.0, 'friendly_name': 'home', 'icon': 'mdi:home'}, 'hass.value': 'zoning', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass'], 'hass.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'FY1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'FY1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "zoning"'}}, 'data': {'hass.domain': 'zone', 'hass.object_id': 'awpc', 'hass.entity_id': 'zone.awpc', 'hass.attributes': {'hidden': True, 'latitude': 33.78268, 'longitude': -84.640061, 'radius': 200.0, 'friendly_name': 'awpc', 'icon': 'mdi:church'}, 'hass.value': 'zoning', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass'], 'hass.geo.location': {'lat': 33.78268, 'lon': -84.640061}}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'Fo1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'Fo1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "zoning"'}}, 'data': {'hass.domain': 'zone', 'hass.object_id': 'work', 'hass.entity_id': 'zone.work', 'hass.attributes': {'hidden': True, 'latitude': 34.062125, 'longitude': -84.252985, 'radius': 200.0, 'friendly_name': 'work', 'icon': 'mdi:office-building'}, 'hass.value': 'zoning', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass'], 'hass.geo.location': {'lat': 12.062125, 'lon': -84.252985}}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'F41ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'F41ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "zoning"'}}, 'data': {'hass.domain': 'zone', 'hass.object_id': 'jackies_work', 'hass.entity_id': 'zone.jackies_work', 'hass.attributes': {'hidden': True, 'latitude': 34.066594, 'longitude': -84.300674, 'radius': 200.0, 'friendly_name': 'jackies_work', 'icon': 'mdi:briefcase'}, 'hass.value': 'zoning', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass'], 'hass.geo.location': {'lat': 12.066594, 'lon': -84.300674}}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'GI1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'GI1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "zoning"'}}, 'data': {'hass.domain': 'zone', 'hass.object_id': 'jackie_gym', 'hass.entity_id': 'zone.jackie_gym', 'hass.attributes': {'hidden': True, 'latitude': 34.082815, 'longitude': -84.355273, 'radius': 200.0, 'friendly_name': 'jackie_gym', 'icon': 'mdi:dumbbell'}, 'hass.value': 'zoning', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass'], 'hass.geo.location': {'lat': 12.082815, 'lon': -84.355273}}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'GY1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'GY1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "zoning"'}}, 'data': {'hass.domain': 'zone', 'hass.object_id': 'airport', 'hass.entity_id': 'zone.airport', 'hass.attributes': {'hidden': True, 'latitude': 33.640728, 'longitude': -84.427696, 'radius': 200.0, 'friendly_name': 'airport', 'icon': 'mdi:airport'}, 'hass.value': 'zoning', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass'], 'hass.geo.location': {'lat': 33.640728, 'lon': -84.427696}}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'Go1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'Go1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "zoning"'}}, 'data': {'hass.domain': 'zone', 'hass.object_id': 'passion_city_church', 'hass.entity_id': 'zone.passion_city_church', 'hass.attributes': {'hidden': True, 'latitude': 33.81696, 'longitude': -84.37083, 'radius': 200.0, 'friendly_name': 'passion_city_church', 'icon': 'mdi:church'}, 'hass.value': 'zoning', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass'], 'hass.geo.location': {'lat': 33.81696, 'lon': -84.37083}}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'HI1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'HI1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "scening"'}}, 'data': {'hass.domain': 'scene', 'hass.object_id': 'weekday_lights_on', 'hass.entity_id': 'scene.weekday_lights_on', 'hass.attributes': {'entity_id': ['switch.family_room_recepticle', 'switch.kitchen_bar_switch', 'switch.butler_pantry_bar', 'switch.kitchen_under_cabinet_sides', 'switch.kitchen_under_cabinet_center', 'switch.kitchen_under_cabinet_right_side', 'switch.living_room_scent_lamp', 'light.breakfast_table_level', 'light.kitchen_ceiling_dimmer'], 'friendly_name': 'weekday_lights_on'}, 'hass.value': 'scening', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'HY1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'HY1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "scening"'}}, 'data': {'hass.domain': 'scene', 'hass.object_id': 'shut_down_all_lights', 'hass.entity_id': 'scene.shut_down_all_lights', 'hass.attributes': {'entity_id': ['group.dining_room_group', 'group.exterior_group', 'group.kitchen_group', 'light.basement_ceiling', 'light.breakfast_table_level', 'light.front_door_chandelier', 'light.hue_lamp_bookcase', 'light.hue_lamp_desk', 'light.hue_lamp_floor', 'light.jackies_office_dimmer', 'light.living_room_chandelier', 'light.yee_light', 'switch.back_yard_landscape_lights', 'switch.back_yard_lights_switch', 'switch.basement_office_fan_light', 'switch.basement_table_lamp', 'switch.basement_tree_light', 'switch.bedroom_hallway', 'switch.family_room_recepticle', 'switch.fireplace', 'switch.living_room_hallway', 'switch.patio_sconce_switch', 'switch.pergola_light_switch', 'switch.unfinished_basement_lights', 'switch.upstairs_hallway'], 'friendly_name': 'shut_down_all_lights'}, 'hass.value': 'scening', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}},
  {'index': {'_index': 'hass-active-v3', '_type': '_doc', '_id': 'Ho1ZC2wBLpP0xwShibWp', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [hass.value] of type [long] in document with id 'Ho1ZC2wBLpP0xwShibWp'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'For input string: "scening"'}}, 'data': {'hass.domain': 'scene', 'hass.object_id': 'back_yard_lights_on_early', 'hass.entity_id': 'scene.back_yard_lights_on_early', 'hass.attributes': {'entity_id': ['switch.back_yard_lights_switch', 'switch.pergola_light_switch', 'switch.back_yard_landscape_lights'], 'friendly_name': 'back_yard_lights_on_early'}, 'hass.value': 'scening', '@timestamp': datetime.datetime(2019, 7, 19, 13, 48, 32, 624752), 'agent.name': 'My Home Assistant', 'agent.type': 'hass', 'agent.version': '0.96.1', 'ecs.version': '1.0.0', 'host.geo.location': {'lat': 12.13596802526734, 'lon': -84.3395096833913}, 'host.architecture': 'x86_64', 'host.os.name': 'Linux', 'host.hostname': 'homeassistant', 'tags': ['hass']}}}
]
legrego commented 5 years ago

Thanks for the update -- that error is interesting. The hass.value field is a multi-field property, which should accept text by default, but then also support hass.value.keyword and hass.value.float if the value is parsable as such:

https://github.com/legrego/homeassistant-elasticsearch/blob/31bc69ba8ee1ff846588642bc8005b21e6ddc89e/custom_components/elastic/index_mapping.json#L27-L39

If you run GET _cat/aliases, do you see an entry for active-hass-index-v3, or similar? I wonder if your installation somehow created that as an index instead of an alias, which could mean your setup didn't use the provided index template which allows for the multi-value setup I described above.

johntdyer commented 5 years ago

@legrego 

root@ubuntu:~# curl http://127.0.0.1:9200/_cat/aliases
metricbeat-7.0.1     metricbeat-7.0.1-2019.07.15-000003 - - -
metricbeat-7.0.0     metricbeat-7.0.0-2019.06.18-000003 - - -
filebeat-7.2.0       filebeat-7.2.0-2019.07.01-000001   - - -
.kibana              .kibana_4                          - - -
metricbeat-7.0.1     metricbeat-7.0.1-2019.06.15-000002 - - -
heartbeat-7.0.1      heartbeat-7.0.1-2019.05.21-000001  - - -
metricbeat-7.1.1     metricbeat-7.1.1-2019.06.21-000001 - - -
heartbeat-7.0.0      heartbeat-7.0.0-2019.06.21-000003  - - -
heartbeat-7.2.0      heartbeat-7.2.0-2019.07.31-000002  - - -
filebeat-7.1.1       filebeat-7.1.1-2019.07.21-000002   - - -
.security            .security-6                        - - -
filebeat-7.0.0       filebeat-7.0.0-2019.07.18-000004   - - -
heartbeat-7.0.1      heartbeat-7.0.1-2019.07.20-000003  - - -
heartbeat-7.1.1      heartbeat-7.1.1-2019.06.24-000001  - - -
metricbeat-7.0.0     metricbeat-7.0.0-2019.04.19-000001 - - -
heartbeat-7.0.1      heartbeat-7.0.1-2019.06.20-000002  - - -
metricbeat-7.2.0     metricbeat-7.2.0-2019.07.01-000001 - - -
filebeat-7.1.1       filebeat-7.1.1-2019.06.21-000001   - - -
heartbeat-7.0.0      heartbeat-7.0.0-2019.04.22-000001  - - -
metricbeat-7.2.0     metricbeat-7.2.0-2019.07.31-000002 - - -
filebeat-7.0.0       filebeat-7.0.0-2019.06.18-000003   - - -
filebeat-7.0.0       filebeat-7.0.0-2019.04.19-000001   - - -
logstash             logstash-2019.06.21-000001         - - -
metricbeat-7.0.0     metricbeat-7.0.0-2019.05.19-000002 - - -
metricbeat-7.1.1     metricbeat-7.1.1-2019.07.21-000002 - - -
heartbeat-7.0.0      heartbeat-7.0.0-2019.07.21-000004  - - -
filebeat-7.2.0       filebeat-7.2.0-2019.07.31-000002   - - -
heartbeat-7.0.0      heartbeat-7.0.0-2019.05.22-000002  - - -
heartbeat-7.2.0      heartbeat-7.2.0-2019.07.01-000001  - - -
active-hass-index-v3 hass-events-v3-000001              - - -
all-hass-events      hass-events-v3-000001              - - -
metricbeat-7.0.0     metricbeat-7.0.0-2019.07.18-000004 - - -
heartbeat-7.1.1      heartbeat-7.1.1-2019.07.24-000002  - - -
filebeat-7.0.1       filebeat-7.0.1-2019.05.16-000001   - - -
filebeat-7.0.0       filebeat-7.0.0-2019.05.19-000002   - - -
metricbeat-7.0.1     metricbeat-7.0.1-2019.05.16-000001 - - -
legrego commented 5 years ago

I stumbled across this too just now when testing something else. My guess is that the component published docs to ES before publishing the corresponding index template/mappings, so ES created one automatically, which ends up being incorrect for subsequent documents that we try to publish.

image

I'll work on mitigating this by ensuring that the index template exists before publishing documents

johntdyer commented 5 years ago

@legrego any ore thoughts on this ?

legrego commented 5 years ago

@johntdyer It's definitely something I want to address, I just haven't had the time recently to fully debug and fix

legrego commented 4 years ago

Beta version is available here which should resolve this. I'll release as the next patch version if all goes well.

legrego commented 4 years ago

Resolved in version 0.2.2