Update for sensor fails after a couple hours

[myopenflixr]

I am having an error that seemed to start on Home Assistant 2023.11. After a couple hours of restarting Home Assistant, the sensors fail and stop updating. The only FIX is to restart Home Assistant to get the sensor data back. But, it stops working and throws errors again after a couple hours.

Here's two entries in the Logs showing the errors.

Any help would be greatly appreciated!

Logger: homeassistant.helpers.entity
Source: helpers/entity.py:696
First occurred: 12:38:35 PM (12 occurrences)
Last logged: 12:44:05 PM

Update for sensor.unifi_gateway_firmware_upgradable fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 696, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 959, in async_device_update
    await hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 190, in wrapper
    result = method(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/unifigateway/sensor.py", line 166, in update
    if devices.get('upgradable'):
       ^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'get'

Logger: homeassistant.helpers.entity
Source: helpers/entity.py:696
First occurred: 12:38:35 PM (60 occurrences)
Last logged: 12:44:05 PM

Update for sensor.unifi_gateway_www fails
Update for sensor.unifi_gateway_wan fails
Update for sensor.unifi_gateway_wlan fails
Update for sensor.unifi_gateway_lan fails
Update for sensor.unifi_gateway_vpn fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 696, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 959, in async_device_update
    await hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 190, in wrapper
    result = method(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/unifigateway/sensor.py", line 178, in update
    if sub['subsystem'] == self._sensor:
       ~~~^^^^^^^^^^^^^
TypeError: string indices must be integers, not 'str'

[PlayFaster]

Hi. Try this, works for me. I have not included the 100ms values, these are the same on my system, I don't think they are valid.

case 'wan':
  # print(h['subsystem'])
  # print(json.dumps(h, indent = 1))
  resp['data'].update({
    "health_" + h['subsystem'] + ".status": h['status'],
    "health_" + h['subsystem'] + ".wan_ip": h['wan_ip'],
    "health_" + h['subsystem'] + ".isp_organization": h['isp_organization'],
    "health_" + h['subsystem'] + ".isp_name": h['isp_name'],
    "health_" + h['subsystem'] + ".gw_version": h['gw_version'],
    "health_" + h['subsystem'] + ".num_sta": h['num_sta'],
    "health_" + h['subsystem'] + ".cpu": h['gw_system-stats']['cpu'],
    "health_" + h['subsystem'] + ".mem": h['gw_system-stats']['mem'],
    "health_" + h['subsystem'] + ".uptime": h['gw_system-stats']['uptime'],
    "health_" + h['subsystem'] + ".tx_bytes": h['tx_bytes-r'],
    "health_" + h['subsystem'] + ".rx_bytes": h['rx_bytes-r'],
    "health_" + h['subsystem'] + ".wan_ip": h['wan_ip'],
    "health_" + h['subsystem'] + ".latency_" + h['uptime_stats']['WAN']['alerting_monitors'][0]['target'] : h['uptime_stats']['WAN']['alerting_monitors'][0]['latency_average'],
    "health_" + h['subsystem'] + ".latency_" + h['uptime_stats']['WAN']['monitors'][0]['target'] : h['uptime_stats']['WAN']['monitors'][0]['latency_average'],
    "health_" + h['subsystem'] + ".latency_" + h['uptime_stats']['WAN']['monitors'][1]['target'] : h['uptime_stats']['WAN']['monitors'][1]['latency_average'],
    "health_" + h['subsystem'] + ".latency_" + h['uptime_stats']['WAN']['monitors'][2]['target'] : h['uptime_stats']['WAN']['monitors'][2]['latency_average'],
    "health_" + h['subsystem'] + ".wan.latency_avg": h['uptime_stats']['WAN']['latency_average']
  })

[coldburn89]

    case 'www':
      # print(json.dumps(h, indent = 1))
      resp['data'].update({
        "health_" + h['subsystem'] + ".status": h['status'],
        "health_" + h['subsystem'] + ".tx_bytes-r": h['tx_bytes-r'],
        "health_" + h['subsystem'] + ".rx_bytes-r": h['rx_bytes-r'],
        "health_" + h['subsystem'] + ".latency": h['latency'],
        "health_" + h['subsystem'] + ".uptime": h['uptime'],
        "health_" + h['subsystem'] + ".drops": h['drops'],
        "health_" + h['subsystem'] + ".xput_up": h['xput_up'],
        "health_" + h['subsystem'] + ".xput_down": h['xput_down'],
        "health_" + h['subsystem'] + ".speedtest_status": h['speedtest_status'],
        "health_" + h['subsystem'] + ".speedtest_lastrun": h['speedtest_lastrun'],
        "health_" + h['subsystem'] + ".speedtest_ping": h['speedtest_ping'],
        "health_" + h['subsystem'] + ".uptime": h['uptime']

      })

Thank you my legend! Its working - awesome!!

[schmierlappe]

is it also possible to get the Alert List into this script? Should be case "alerts"

[coldburn89]

i am also really curious how to extract more like from this url: "https://192.168.1.1/proxy/network/api/s/default/stat/device" and maybe something to monitor upgrades. Also good to sort the attributes like WAN, WWW,WLAN etc instead of one bunch into data. Maybe a "feature request" for this fan made integration / repair of the original 😄

BUT i am already very happy with the progress and that its working like a charm!! Again, i am very greatfull for the help! 🍻

[PlayFaster]

Hi @coldburn89 . On my system, the "https://xxx.xxx.xxx.xxx/proxy/network/api/s/default/stat/device" URL contains 26,123 (26K) data pairs!

From what I can see it's info on every device - console, switch, switch-ports, access points, clients and config info - on the system. Useful perhaps, but also very unwieldy (because of overall size). For me, I cannot see additional info in here that I want to track, not to say it would not be useful to yourself and others.

Note that this source is where johntdyer is pulling the unifi device info in his script from, the info for the UniFi console, APs and Switches. Its just not pulling all of the additional info.

Also, I'd note that some of the additional info, esp. on clients, can be obtained by selecting the additional options in the configuration for the official HA UniFi integration (i.e. uptime and bandwidth sensors).

Is there specific additional info that you see in the source URL, not available from the official integration, that you want? If so, call it out and I’ll try to take a look - there’s certainly enough info for there to be other parts of interest, I just can’t see them at the moment.

[PlayFaster]

Hi @coldburn89 . RE: Updates and Upgrades ...

For me the script returns an Update attribute line for each tracked device (i.e. console, switch, AP). So for example I have :

udm_pro.Update: 0
uni_fi_switch24_g_g2.Update: 0

... plus others. I have no updates pending right now but I assume (hope) that these would turn from 0 to 1 if there was an update pending.

I'm not sure if this is what you were referring to in relation to "something to monitor upgrades". ? The updates are in there (I think).

The only thing that this will give you that the official HA integration does not is updates pending on the UDM Pro itself. I've only ever seen the official integration show updates for devices like switches and APs.

Not sure yet whether the UDM Update flag will show app updates (like UniFi network) and/or OS updates. Probably app only I think, based on how the Gateway module used to work.

Maybe I’m misinterpreting, let me know. For me, almost all of this functionality is already in the official HA integration, although I have added one sensor for the UDM Pro update itself.

As you have noted, there is some functionality in the script that probably should be a “feature request” for the official integration.

[PlayFaster]

Hi @schmierlappe . RE: Alert Info

The event log does seem to be accessible, via ...

 https://your.console.url/proxy/network/api/s/default/stat/event

This holds the last 3,000 (3K) events, all types. I assume it would be possible to parse this into an attribute list, similar to the script. Probably best to have it as a separate HA command line sensor, as the payload (attribute list) size will be large.

Not sure how to implement this, but there are certainly some elements of the johntdyer script that might be repurposed. I'm tied up for the next two days, but might try to have a look at the weekend, as there is some info in the event log list that I would find useful in HA. If anyone else knows how to make this happen - please have at it!

REFERENCE - URL LIST For reference, the list of API URLs that I have found to work on a UDM Pro, from which info can be parsed, is ...

High Level Info
 https://your.console.url/proxy/network/api/s/default/stat/sysinfo

High Level Info incl Latency
 https://your.console.url/proxy/network/api/s/default/stat/health

All Device incl Client Info
 https://your.console.url/proxy/network/api/s/default/stat/device

Detailed Client Info
 https://your.console.url/proxy/network/api/s/default/stat/sta

Event Log
 https://your.console.url/proxy/network/api/s/default/stat/event

[coldburn89]

@PlayFaster currently I don't have anything to ask for. Only thing we could check is if we could seperate some things from data and divide them. But that's a nice to have and maybe for the creator of the integration.

Do you have things you want to monitor? I think the current config suits my needs. Again, I am grateful for the help of this community!

[plouf34]

Hello Someone could summarize how to fix the stop integration after a couple of hours?

[PlayFaster]

Hello Someone could summarize how to fix the stop integration after a couple of hours?

Hi. In summary, there is no fix for this issue. The UniFigateWay module is failing and is no longer being maintained it seems.

What is available, if you search thru this issue, is a script provided by johntdyer that can provide very similiar functionality.

[PlayFaster]

is it also possible to get the Alert List into this script? Should be case "alerts"

Hi @schmierlappe . I have tried to access the alert info and I am not able to. Trying an "alerts" URL does not work for me, on my UDM Pro. Neither does "alert" or "critical" or several other variations.

The "events" URL does work, however this provides the CLIENT section of the System Log from the UniFi Web Page - info on connects and disconnects. On my system this does not provide any of the more important Alert type events.

So, a no-go for me I'm afraid, as I can't find a way to access the alerts.

[PlayFaster]

@PlayFaster currently I don't have anything to ask for. Only thing we could check is if we could seperate some things from data and divide them. But that's a nice to have and maybe for the creator of the integration.

Do you have things you want to monitor? I think the current config suits my needs. Again, I am grateful for the help of this community!

Hi @coldburn89 . Agreed. Ideally, I would like to see all of the items from the original UniFiGateWay module available from the official Unifi Network integration, but as I have access to them via the script, its not a major priority.

That said, what would be most useful for me would be having some of the key alerts from Unifi in HA, especially ISP alerts - i.e. high latency or failover.

[KosieKramer]

I have chosen another route that works for me getting data from my UDM. It uses no script files or third-party components (except curl of course).

In configuration.yaml:

command_line:
  - sensor:
      name: Unifi Raw
      command: 'curl -o /dev/null -H ''Content-Type: application/json'' -d ''{"username": "XXXX", "password": "XXXX"}'' -ks https://XXXX/api/auth/login -c cookies.txt && curl -ks https://XXXX/proxy/network/api/s/default/stat/device -b cookies.txt'
      value_template: "{{ 'OK' }}"
      json_attributes:
        - data
      scan_interval: 30

template:
  - name: UDM Board Temperature
    state: "{{ state_attr('sensor.unifi_raw', 'data')[0]['temperatures'][0].value }}"
    unit_of_measurement: °C
    state_class: measurement

  - name: UDM CPU Temperature
    state: "{{ state_attr('sensor.unifi_raw', 'data')[0]['temperatures'][1].value }}"
    unit_of_measurement: °C
    state_class: measurement

  - name: "UDM Processor Use"
    state: "{{ state_attr('sensor.unifi_raw', 'data')[0]['system-stats'].cpu }}"
    unit_of_measurement: "%"
    state_class: measurement

  - name: "UDM Memory Used Percent"
    state: "{{ state_attr('sensor.unifi_raw', 'data')[0]['system-stats'].mem }}"
    unit_of_measurement: "%"
    state_class: measurement

  - name: "UDM WAN Download Speed"
    state: "{{ (state_attr('sensor.unifi_raw', 'data')[0]['uplink']['rx_bytes-r'] / 1024 / 1024) | float(2) }}"
    unit_of_measurement: "MB/s"
    state_class: measurement

  - name: "UDM WAN Upload Speed"
    state: "{{ (state_attr('sensor.unifi_raw', 'data')[0]['uplink']['tx_bytes-r'] / 1024 / 1024) | float(2) }}"
    unit_of_measurement: "MB/s"
    state_class: measurement

Replace the XXXX with your relevant info and off you go!

There is plenty of room for refinement and templating more info from the attributes.

[sumofatguy]

I have chosen another route that works for me getting data from my UDM. It uses no script files or third-party components (except curl of course).

In configuration.yaml:

command_line:
  - sensor:
      name: Unifi Raw
      command: 'curl -o /dev/null -H ''Content-Type: application/json'' -d ''{"username": "XXXX", "password": "XXXX"}'' -ks https://XXXX/api/auth/login -c cookies.txt && curl -ks https://XXXX/proxy/network/api/s/default/stat/device -b cookies.txt'
      value_template: "{{ 'OK' }}"
      json_attributes:
        - data
      scan_interval: 30

template:
  - name: UDM Board Temperature
    state: "{{ state_attr('sensor.unifi_raw', 'data')[0]['temperatures'][0].value }}"
    unit_of_measurement: °C
    state_class: measurement

  - name: UDM CPU Temperature
    state: "{{ state_attr('sensor.unifi_raw', 'data')[0]['temperatures'][1].value }}"
    unit_of_measurement: °C
    state_class: measurement

  - name: "UDM Processor Use"
    state: "{{ state_attr('sensor.unifi_raw', 'data')[0]['system-stats'].cpu }}"
    unit_of_measurement: "%"
    state_class: measurement

  - name: "UDM Memory Used Percent"
    state: "{{ state_attr('sensor.unifi_raw', 'data')[0]['system-stats'].mem }}"
    unit_of_measurement: "%"
    state_class: measurement

  - name: "UDM WAN Download Speed"
    state: "{{ (state_attr('sensor.unifi_raw', 'data')[0]['uplink']['rx_bytes-r'] / 1024 / 1024) | float(2) }}"
    unit_of_measurement: "MB/s"
    state_class: measurement

  - name: "UDM WAN Upload Speed"
    state: "{{ (state_attr('sensor.unifi_raw', 'data')[0]['uplink']['tx_bytes-r'] / 1024 / 1024) | float(2) }}"
    unit_of_measurement: "MB/s"
    state_class: measurement

Replace the XXXX with your relevant info and off you go!

There is plenty of room for refinement and templating more info from the attributes.

Worked great until I added a new Ubiquiti device on my network and my udmp moved to data[1] instead of data[0]

[edmund-1]

I did.... So try these steps

1. Take the python code and paste that into /config/scripts/unifi_combined.py
2. chmod +x /config/scripts/unifi_combined.py
3. Add the following to your configuration.yaml

command_line:

  - sensor:
      command: '/config/scripts/unifi_combined.py'
      command_timeout: 180
      name: unifi_stats
      value_template: '{{value_json.version}}' 
      scan_interval: 300
      json_attributes:
        - data

4. run the command_line.reload service from developer tools
5. Goto entities and look for sensor.unifi_stats

Hope that helps

Hi folks,

Thank you @johntdyer and @PlayFaster for your guide. I am using UDMPRO and HA on Raspberry Pi5.

I tried to implement @johntdyer's approach; however, I encounter the following issues:

1. After HA reboot the module requested is gone. I need to run pip3 install requests after each reboot.
2. In HA logs, execution for /config/scripts/unifi_combined.py fails with error 127

   
   Logger: homeassistant.components.command_line.utils
   Source: components/command_line/utils.py:55
   integration: Command Line ([documentation](https://www.home-assistant.io/integrations/command_line), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+command_line%22))
   First occurred: 22:23:00 (9 occurrences)
   Last logged: 23:03:00

Command failed (with return code 127): /config/scripts/unifi_combined.py

Checked python variable with `which python3` and updated script `/config/scripts/unifi_combined.py` with `#!/usr/bin/python3`

3. `python3 /config/scripts/unifi_combined.py` reads the attributes, and have some InsecureRequestsWarning, expected as `verify=False`
![image](https://github.com/custom-components/sensor.unifigateway/assets/134559937/5371ffaa-da4a-436e-9696-539d5613a08e)
...
![image](https://github.com/custom-components/sensor.unifigateway/assets/134559937/f24d83e8-60c0-4669-8c29-0b5f931449e0)

4. even if the attributes are displayed after executing `python3 /config/scripts/unifi_combined.py` sensor.unifi_stats entity has unknown status.
![image](https://github.com/custom-components/sensor.unifigateway/assets/134559937/a2666993-c166-4bc1-b236-1a3549c7fa2e)

Most probably I did something wrong, somewhere. What am I doing wrong? Can you please guide me?

Thank you.