Version 0.6.5 - IndexError: list index out of range

[b-per]

Hi,

I am using the 0.6.5 version on my Raspberry Pi 3 running Home Assistant 0.108.3 on docker. In some occasions, the temperature sensor stops working and I usually restart my Pi to fix the problem.

Overnight here is the issue that I found in the logs. After that time the temp sensor has stopped updating.

2020-04-17 00:00:00 ERROR (MainThread) [homeassistant.core] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/mitemp_bt/sensor.py", line 689, in update_ble
    discover_ble_devices(config, aeskeys)
  File "/config/custom_components/mitemp_bt/sensor.py", line 670, in discover_ble_devices
    sensors[f_i], mac, config, formaldehyde_m_data[mac], fdec=3
IndexError: list index out of range

I'm happy to provide additional inputs if required.

[Magalex2x14]

Hello! Thanks for reporting. What sensors do you use? Is there a formaldehyde sensor among them?

[Magalex2x14]

Please also show the contents of the file <HA folder>/custom_components/mitemp_bt/const.py and try to reinstall component.

[b-per]

Hi! I have in my apartment 1 temp/humidity sensor LYWSDCGQ and 3 Mi Flora HHCCJCY01, but those 3 plant sensors are still using the default miflora integration in Home Assistant (and they are actually still updating the values now even if LYWSDCGQ stopped after the error).

Here is the content of const.py:

"""Constants for the Xiaomi passive BLE monitor sensor integration."""

# Configuration options
CONF_ROUNDING = "rounding"
CONF_DECIMALS = "decimals"
CONF_PERIOD = "period"
CONF_LOG_SPIKES = "log_spikes"
CONF_USE_MEDIAN = "use_median"
CONF_ACTIVE_SCAN = "active_scan"
CONF_HCI_INTERFACE = "hci_interface"
CONF_BATT_ENTITIES = "batt_entities"
CONF_ENCRYPTORS = "encryptors"
CONF_REPORT_UNKNOWN = "report_unknown"

# Default values for configuration options
DEFAULT_ROUNDING = True
DEFAULT_DECIMALS = 1
DEFAULT_PERIOD = 60
DEFAULT_LOG_SPIKES = False
DEFAULT_USE_MEDIAN = False
DEFAULT_ACTIVE_SCAN = False
DEFAULT_HCI_INTERFACE = 0
DEFAULT_BATT_ENTITIES = False
DEFAULT_REPORT_UNKNOWN = False

"""Fixed constants."""

# Sensor measurement limits to exclude erroneous spikes from the results
CONF_TMIN = -40.0
CONF_TMAX = 60.0
CONF_HMIN = 0.0
CONF_HMAX = 99.9

# Xiaomi sensor types dictionary for adv parser
XIAOMI_TYPE_DICT = {
    b'\x98\x00': "HHCCJCY01",
    b'\xAA\x01': "LYWSDCGQ",
    b'\x5B\x04': "LYWSD02",
    b'\x47\x03': "CGG1",
    b'\x5D\x01': "HHCCPOT002",
    b'\xBC\x03': "GCLS002",
    b'\x5B\x05': "LYWSD03MMC",
    b'\x76\x05': "CGD1",
    b'\xDF\x02': "JQJCY01YM"
}

# Sensor type indexes dictionary
# Temperature, Humidity, Moisture, Conductivity, Illuminance, Formaldehyde, Battery
# Measurement type T  H  M  C  I  F  B   9 - no measurement
MMTS_DICT = {
    'HHCCJCY01' : [0, 9, 1, 2, 3, 9, 9],
    'GCLS002'   : [0, 9, 1, 2, 3, 9, 9],
    'HHCCPOT002': [9, 9, 0, 1, 9, 9, 9],
    'LYWSDCGQ'  : [0, 1, 9, 9, 9, 9, 2],
    'LYWSD02'   : [0, 1, 9, 9, 9, 9, 9],
    'CGG1'      : [0, 1, 9, 9, 9, 9, 2],
    'LYWSD03MMC': [0, 1, 9, 9, 9, 9, 2],
    'CGD1'      : [0, 1, 9, 9, 9, 9, 2],
    'JQJCY01YM' : [0, 1, 9, 9, 9, 2, 3]
}

I'll delete the component from HACS and reinstall it as well.

[Magalex2x14]

Hmm... I have no explanation yet. I can’t imagine a situation that could lead to this error...

If reinstalling the component did not help, then replace the contents of the sensor.py file with the contents of the attached file: sensor.py.txt reboot HA, wait for the error in the log with the text "IndexError!", and show the next entry after it.

[b-per]

I just restarted HA and it's working again. It usually takes a few hours or even days to before it fails but I will keep you posted if it happens again, and I'll try your new sensor file then.

[Magalex2x14]

OK, thank you.

[Magalex2x14]

I found where I remembered your nickname from - it was a strange problem with malformed/corrupted messages #12 I suspect that now we could run into something similar. Thereat the issue was resolved by tightening the verification of messages for validity, and it is likely that now we are faced with something that passed through this verification.

Now I have a deep night. Tomorrow I will prepare the code for deeper debugging because I suspect that the code from my previous post will not clarify the situation...

[b-per]

It just failed again after only being up for 2 hours. The error is the exact same.

I am going to try with the code you provided earlier and see if there is more details coming up in the logs.

[Magalex2x14]

@camfrout I forgot to mention that the code from this file will no longer drop out with an error - it will just write to the log, so if you wait for the component to fail, then do not wait, just look from time to time in the log...

[b-per]

OK. It hasn't failed yet and there is no error in the logs so far, but I'll keep you posted.

[b-per]

Hi! The error showed twice in the past 6 hours. Here is the output.

First one:

2020-04-18 14:22:05 ERROR (SyncWorker_15) [custom_components.mitemp_bt.sensor] IndexError!
2020-04-18 14:22:05 ERROR (SyncWorker_15) [custom_components.mitemp_bt.sensor] f_i = 9, data = [4.92]

Second one:

2020-04-18 16:48:30 ERROR (SyncWorker_10) [custom_components.mitemp_bt.sensor] IndexError!
2020-04-18 16:48:30 ERROR (SyncWorker_10) [custom_components.mitemp_bt.sensor] f_i = 9, data = [5.28]

I don't own any Formaldehyde sensor but I live in a building with many units and sometimes can see Bluetooth devices from my neighbours.

[Magalex2x14]

OK. Thank you. I prepared a new debugging code for your situation: sensor.py.txt As before, it should not fail, but there will be more detailed messages in the log. Please try it.

[b-per]

Hi! The new version is failing though, unfortunately. I got the following error in the logs and the component stopped working after.

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/mitemp_bt/sensor.py", line 697, in update_ble
    discover_ble_devices(config, aeskeys)
  File "/config/custom_components/mitemp_bt/sensor.py", line 508, in discover_ble_devices
    data = parse_raw_message(msg, aeskeyslist, report_unknown)
  File "/config/custom_components/mitemp_bt/sensor.py", line 329, in parse_raw_message
    res, mblogged = parse_xiaomi_value(xvalue, xvalue_typecode)
TypeError: cannot unpack non-iterable NoneType object

[Magalex2x14]

Oh, yes, I see... Please, try this one: sensor.py.txt

[b-per]

Hi again. Here is the error:

2020-04-19 13:29:40 ERROR (SyncWorker_19) [custom_components.mitemp_bt.sensor] Formaldehyde message: 043e23020100003e79d8a8654c17020106131695fe5020aa01443e79d8a8654c1000023102b3
2020-04-19 13:29:40 ERROR (SyncWorker_19) [custom_components.mitemp_bt.sensor] Current result dict: {'rssi': -77, 'mac': '4C65A8D8793E', 'type': 'LYWSDCGQ', 'packet': 68, 'formaldehyde': 5.61}
2020-04-19 13:29:40 ERROR (SyncWorker_19) [custom_components.mitemp_bt.sensor] Parsed formaldehyde data: {'rssi': -77, 'mac': '4C65A8D8793E', 'type': 'LYWSDCGQ', 'packet': 68, 'formaldehyde': 5.61}

Unless you need more debug log I might go back to the previous version that was not breaking the component when there was an error. With today's version the component broke after the error.

[Magalex2x14]

Your last log is enough for now. Give me some time - I'll try to figure out what's going on here...

[Magalex2x14]

OK. I published 0.6.6-beta, designed to solve your problem.

Please try it. You can install using HACS by turning on betas:

[b-per]

Thanks! I'll give it a go and will let you know.

[b-per]

Quick update to let you know that the component has been working with no issue for 2 days now. Thanks for the fix in the beta! 👍

[Magalex2x14]

OK. Thank you. I think that this weekend I will release.

[Magalex2x14]

0.6.6 released. Feel free to reopen this issue, if needed.