filippopetri
commented
2 months ago Same problem after updating
Closed Stephan-4711 closed 2 months ago
filippopetri
commented
2 months ago Same problem after updating
alexdelprete
commented
2 months ago please enable debug on the integration and post the debug log.
Stephan-4711
commented
2 months ago I'm just on my mobile until Friday. No way to do this, maybe @filippopetri?
filippopetri
commented
2 months ago The relevant part is:
2024-09-04 16:18:28.377 DEBUG (SyncWorker_24) [custom_components.opnsense] execution time: OPNSenseData._get_system_info 0.053
2024-09-04 16:18:28.434 DEBUG (SyncWorker_24) [custom_components.opnsense] execution time: OPNSenseData._get_host_firmware_version 0.057
2024-09-04 16:18:28.572 DEBUG (SyncWorker_24) [custom_components.opnsense] execution time: OPNSenseData._get_firmware_update_info 0.138
2024-09-04 16:18:28.672 DEBUG (SyncWorker_24) [custom_components.opnsense.pyopnsense] [get_telemetry_interfaces] interface_info: {'status': 403, 'message': 'Forbidden'}
2024-09-04 16:18:28.752 DEBUG (SyncWorker_24) [custom_components.opnsense.pyopnsense] [get_telemetry_mbuf] mbuf_info: {'status': 403, 'message': 'Forbidden'}
2024-09-04 16:18:28.752 DEBUG (SyncWorker_24) [custom_components.opnsense.pyopnsense] [get_telemetry_mbuf] mbuf: {'used': None, 'total': None, 'used_percent': None}
2024-09-04 16:18:28.823 DEBUG (SyncWorker_24) [custom_components.opnsense.pyopnsense] [get_telemetry_pfstate] pfstate_info: {'status': 403, 'message': 'Forbidden'}
2024-09-04 16:18:28.823 DEBUG (SyncWorker_24) [custom_components.opnsense.pyopnsense] [get_telemetry_pfstate] pfstate: {'used': None, 'total': None, 'used_percent': None}
2024-09-04 16:18:28.875 DEBUG (SyncWorker_24) [custom_components.opnsense.pyopnsense] [get_telemetry_memory] memory_info: {'status': 403, 'message': 'Forbidden'}
2024-09-04 16:18:28.920 ERROR (SyncWorker_24) [custom_components.opnsense.pyopnsense] Unexpected _get_telemetry_memory error err=IndexError('list index out of range'), type(err)=<class 'IndexError'>
2024-09-04 16:18:28.920 ERROR (SyncWorker_24) [custom_components.opnsense.pyopnsense] Unexpected get_telemetry error err=IndexError('list index out of range'), type(err)=<class 'IndexError'>
2024-09-04 16:18:28.920 ERROR (MainThread) [custom_components.opnsense] Unexpected error fetching OPNsense OPNsense state data
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 354, in _async_refresh
self.data = await self._async_update_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 271, in _async_update_data
return await self.update_method()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/init.py", line 97, in async_update_data
await hass.async_add_executor_job(lambda: data.update())
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/init.py", line 97, in
IndexError: list index out of range
2024-09-04 16:18:28.922 DEBUG (MainThread) [custom_components.opnsense] Finished fetching OPNsense OPNsense state data in 0.598 seconds (success: False)
Tell me if you need more logsLooks like a permission issue:
2024-09-04 16:18:28.875 DEBUG (SyncWorker_24) [custom_components.opnsense.pyopnsense] [get_telemetry_memory] memory_info: {'status': 403, 'message': 'Forbidden'}
alexdelprete
commented
2 months ago These are the privileges required by the REST API endpoints implemented by @Snuffy2 in PR #147 :
| Endpoint | Permission |
|---|---|
| /api/interfaces/overview/export | Status: Interfaces |
| /api/diagnostics/system/system_mbuf | Dashboard (all) |
| /api/diagnostics/firewall/pfstates | Dashboard (all) |
| /api/diagnostics/system/systemResources | Dashboard (all) |
| /api/diagnostics/system/systemSwap | Dashboard (all) |
| /api/diagnostics/system/systemTime | Lobby: Login / Logout / Dashboard |
| /api/diagnostics/cpu_usage/getCPUType | Dashboard (all) |
| /api/diagnostics/system/systemDisk | Dashboard (all) |
| /api/openvpn/export/providers | VPN: OpenVPN: Client Export Utility |
| /api/openvpn/service/searchSessions | Status: OpenVPN |
| /api/routes/gateway/status | Dashboard (all) |
filippopetri
commented
2 months ago I updated the privileges as requested but the integration is still not connecting. This time the debug logs are:
2024-09-04 16:53:36.629 ERROR (SyncWorker_4) [custom_components.opnsense.pyopnsense] Unexpected get_system_info error err=TypeError("string indices must be integers, not 'str'"), type(err)=<class 'TypeError'>
2024-09-04 16:53:36.630 ERROR (MainThread) [custom_components.opnsense] Unexpected error fetching OPNsense OPNsense state data
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 354, in _async_refresh
self.data = await self._async_update_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 271, in _async_update_data
return await self.update_method()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/init.py", line 97, in async_update_data
await hass.async_add_executor_job(lambda: data.update())
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/init.py", line 97, in
TypeError: string indices must be integers, not 'str'
2024-09-04 16:53:36.634 DEBUG (MainThread) [custom_components.opnsense] Finished fetching OPNsense OPNsense state data in 0.051 seconds (success: False)I added the opnsense user to the admin group and now it's connected
Stephan-4711
commented
2 months ago Admin rights seems to be a little to much 🤯
filippopetri
commented
2 months ago Admin rights seems to be a little to much 🤯
I agree, but with the new requested permissions the integration cannot connect. I tried adding the user to the admin group to test where the problem was and now everything works
alexdelprete
commented
2 months ago Admin rights seems to be a little to much 🤯
I agree, but with the new requested permissions the integration cannot connect. I tried adding the user to the admin group to test where the problem was and now everything works
It's not a "connection" issue, it's a privilege issue: probably some more privilege is required, that's why admin works. For now, use admin, then we'll have to find which privilege is needed.
Please post the debug log lines in which you find error 403, or post the entire log.
alexdelprete
commented
2 months ago Admin rights seems to be a little to much 🤯
in the end, adding all those privileges granularly will be almost equivalent to the admin profile.
alexdelprete
commented
2 months ago @filippopetri please take into account that the new REST API functionality requires the privileges mentioned in the previous table, but we use xmlrpc for other data, so the privileges described in the readme are to be left in place (system: firmware and XMLrpc). Did you leave those two in addition to the above privilege table?
alexdelprete
commented
2 months ago I updated documentation with all required privileges, if someone can test this please give me a feedback:
Stephan-4711
commented
2 months ago I guess there must be still something missing. I added all these permissions, see the log:
`2024-09-04 21:11:13.220 DEBUG (SyncWorker_41) [custom_components.opnsense.pyopnsense] [get_telemetry_pfstate] pfstate_info: {'status': 403, 'message': 'Forbidden'}
2024-09-04 21:11:13.220 DEBUG (SyncWorker_41) [custom_components.opnsense.pyopnsense] [get_telemetry_pfstate] pfstate: {'used': None, 'total': None, 'used_percent': None}
2024-09-04 21:11:13.351 DEBUG (SyncWorker_41) [custom_components.opnsense.pyopnsense] [get_telemetry_memory] memory_info: {'memory': {'total': '16930144256', 'total_frmt': '16145', 'used': 3422065066, 'used_frmt': '3263', 'arc': '2180078184', 'arc_frmt': '2079', 'arc_txt': 'ARC size 2079 MB'}}
2024-09-04 21:11:13.429 ERROR (SyncWorker_41) [custom_components.opnsense.pyopnsense] Unexpected _get_telemetry_memory error err=IndexError('list index out of range'), type(err)=<class 'IndexError'>
2024-09-04 21:11:13.429 ERROR (SyncWorker_41) [custom_components.opnsense.pyopnsense] Unexpected get_telemetry error err=IndexError('list index out of range'), type(err)=<class 'IndexError'>
2024-09-04 21:11:13.430 ERROR (MainThread) [custom_components.opnsense] Unexpected error fetching firewall.domain.com OPNsense state data
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 354, in _async_refresh
self.data = await self._async_update_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 271, in _async_update_data
return await self.update_method()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/init.py", line 97, in async_update_data
await hass.async_add_executor_job(lambda: data.update())
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/init.py", line 97, in
IndexError: list index out of range`Looks like it's pfstate: I'll check opnsense docs for relevant privileges. Are you positive that you set ALL required privileges?
2024-09-04 21:11:13.220 DEBUG (SyncWorker_41) [custom_components.opnsense.pyopnsense] [get_telemetry_pfstate] pfstate_info: {'status': 403, 'message': 'Forbidden'} 2024-09-04 21:11:13.220 DEBUG (SyncWorker_41)I double checked there is no pfsfate, all permissions you posted I added to the group
alexdelprete
commented
2 months ago I double checked there is no pfsfate
I meant it's the pfstate API call that returns 403 (not authorized):
pfstate_info: dict[str, Any] = self._post("/api/diagnostics/firewall/pfstates")
Snuffy2
commented
2 months ago Ok, catching up on everything in the thread.... some responses.
File "/config/custom_components/opnsense/pyopnsense/init.py", line 1005, in _get_telemetry_memory
or not isinstance(swap_info.get("swap", [])[0], Mapping)This is an error when getting the Memory details. I'll put out a fix so that if this does happen it handles this gracefully and shows a better error.
File "/config/custom_components/opnsense/pyopnsense/init.py", line 135, in _exec_php
response = json.loads(response["real"])
~~~~~~~~^^^^^^^^
TypeError: string indices must be integers, not 'str'This is unrelated to the changes in this release and due to permissions as you guys determined. I may still make some more changes so it is handled more gracefully and shows a better error.
Let me investigate the pfstates and permissions issue. Despite it not getting pfstates data, that section doesn't appear to be throwing an error.
Admin rights seems to be a little too much 🤯
This integration essentially needs full read-only access to Opnsense plus some write permissions in order to change these items. However, there isn't really a full read-only access that I'm aware of. Most of the permissions include both read and write access to the various sections. In addition, the XMLRPC permission essentially gives admin permissions already as you can do pretty much everything using XMLRPC if you know how to. We may be transitioning more from XMLRPC to the published REST API as it is better supported. As this happens, we may ultimately need to give admin access to this integration.
alexdelprete
commented
2 months ago This is an error when getting the Memory details. I'll put out a fix so that if this does happen it handles this gracefully and shows a better error.
When using admin privileges, the error goes away. I was thinking: is there a way to check for permissions of the user before executing the GET/POST?
We may be transitioning more from XMLRPC to the published REST API as it is better supported
Franco (OPNsense dev), told me and Travis when we started this integration, that XMLrpc will be probably removed in the future and they will only support the API.
If you agree, I would simply start now with the admin privilege prerequirement for the integration to work with the API, so we'll get rid of a LOT of future issues. What do you think?
alexdelprete
commented
2 months ago @Stephan-4711 I merged @Snuffy2's PR #149 in a beta (v0.1.23-beta.1). You need to enable beta versions access in HACS in order to install it from there, could you tell me if it works with the required privileges listed in the readme?
Thanks a lot.
Snuffy2
commented
2 months ago @Stephan-4711 I merged @Snuffy2's PR #149 in a beta (
v0.1.23-beta.1). You need to enable beta versions access in HACS in order to install it from there, could you tell me if it works with the required privileges listed in the readme?Thanks a lot.
filippopetri
commented
2 months ago Thank you everyone! Will test as soon as PR #150 it's ready
Stephan-4711
commented
2 months ago First try shows the same error. Currently no time for a debug log. Maybe this evening ECT
buzzard10
commented
2 months ago Hello Guys,
i don't know whether it's related but i just started with Opnsense and i can't connect HASS to Opnsense even for the first time.
I have this error:
I found in logs this:
source: custom_components/opnsense/config_flow.py:149
Unexpected err=AbortFlow('Flow aborted: already_configured'), type(err)=<class 'homeassistant.data_entry_flow.AbortFlow'>
Unexpected err=AbortFlow('Flow aborted: already_in_progress'), type(err)=<class 'homeassistant.data_entry_flow.AbortFlow'>@Stephan-4711 @filippopetri v0.2.0-beta.1 has been released. Please test it and give us a feedback.
Take into account that the official prerequirement from now on is that the user has to have admin privileges. You can test granular permissions if you want, but we don't officially recommend it.
filippopetri
commented
2 months ago I'm using admin privileges and everything is looking good
alexdelprete
commented
2 months ago I'm using admin privileges and everything is looking good
thanks for the feedback.
alexdelprete
commented
2 months ago I created a thread in the Discussions section, let's continue there, hope to receive other positive/negative feedbacks.
I'll close this issue.
Stephan-4711
commented
2 months ago I'm using WITHOUT ADMIN privileges, seems to work too for now
alexdelprete
commented
2 months ago I'm using WITHOUT ADMIN privileges, seems to work too for now
It should work, if you configure it with all privileges mentioned in the readme. Problem is that we can't guarantee it works in the future while we expand the REST API usage, so we decided to officially recommend using an admin user.
Thanks for the feedback.
I updated to the last version. Still get error while loading the integration. I removed everything from the integration also from filesystem.
`Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht
Logger: custom_components.opnsense Quelle: helpers/update_coordinator.py:354 Integration: OPNsense (Dokumentation, Probleme) Erstmals aufgetreten: 15:37:22 (16 Vorkommnisse) Zuletzt protokolliert: 15:52:05
Unexpected error fetching firewall name OPNsense state data Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 354, in _async_refresh self.data = await self._async_update_data() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 271, in _async_update_data return await self.update_method() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/opnsense/init.py", line 97, in async_update_data await hass.async_add_executor_job(lambda: data.update()) File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/opnsense/init.py", line 97, in
await hass.async_add_executor_job(lambda: data.update())
^^^^^^^^^^^^^
File "/config/custom_components/opnsense/init.py", line 316, in update
self._state["telemetry"] = self._get_telemetry()
^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/init.py", line 225, in inner
response = func(*args, *kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/init.py", line 249, in _get_telemetry
return self._client.get_telemetry()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/pyopnsense/init.py", line 104, in inner
raise err
File "/config/custom_components/opnsense/pyopnsense/init.py", line 101, in inner
return func(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/pyopnsense/init.py", line 871, in get_telemetry
telemetry["memory"] = self._get_telemetry_memory()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/pyopnsense/init.py", line 104, in inner
raise err
File "/config/custom_components/opnsense/pyopnsense/init.py", line 101, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/opnsense/pyopnsense/init.py", line 1005, in _get_telemetry_memory
or not isinstance(swap_info.get("swap", [])[0], Mapping)