search

tomaae / homeassistant-mikrotik_router

Mikrotik router integration for Home Assistant
Apache License 2.0
301 stars 50 forks source link

[Enhancement] [v7.1beta] Migrate from Accounting to Traffic Flow #61

Closed jwshields closed 3 years ago

jwshields commented 4 years ago

I am not sure what is going on here. But it seems that this integration is not working as it should. All of my devices are being marked as "unavailable" in HA (See attached screenshot) But it seems they are also, momentarily (I assume this is when the scanning is happening) - they do appear, but they quickly go back to being unavailable.

How to reproduce the issue

  1. Configure integration in HA WebUI
  2. Add discovered devices

Expected behavior

Devices will appear available and tracked properly, counters & gauges tracked properly as well.

Screenshots

image image image image

Software versions

Traceback/Error logs

Debug log of integration, https://gist.github.com/jwshields/eda18b7c64684cf243ac1561953ceadc

Additional context

Not sure if I am missing any configurations or settings here, but any help would be appreciated.

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.97. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

tomaae commented 4 years ago

There are 2 problems.

  1. You are missing language files, looks like something during HACS installation went wrong. Try removing the component and install it again.
  2. Loop crashes on "error while building list for path : unknown". Seems to be related to accounting in beta version you are using. I dont use development branch even on test machines because rollback can be problematic. I will have a look at 7.1 when my workload lowers a bit, to see what they changed.
jwshields commented 4 years ago

Thanks for the response...

WRT 1. I think that's because I selected master in HACs, I changed the version of mikrotik_router back to v1.5.10 and the options appears to be proper once more

  1. Poking around my environment in my router, I do not see any settings for accounting (When I was on 6.41.1 I did not use accounting any way)

The core problem I am thinking is that the sensors and devices are not being tracked properly. Does that have to do with accounting? In the debug logs, I can see lots of information being given, but, the devices themselves are showing up intermittently as shown in my screenshots.

tomaae commented 4 years ago

ah yes, master is not compiled. that would explain the issue.

it crashes on accounting. not sure if they moved it to extra package now

alexwat29 commented 3 years ago

same problem, plugin didnt works with routeOS7

Robbo312 commented 3 years ago

From MikroTik Forum;.

IP(v4) Accounting has been removed in RouterOS v7. So certainly there has been no IPv6 Accounting added. Accounting should now be done using Netflow (IP->Traffic Flow) which already supports IPv6 in RouterOS v6.

https://forum.mikrotik.com/viewtopic.php?p=800360#p803180

tomaae commented 3 years ago

Ok, so we will have to rework accounting to traffic flow. @ivanpavlina What do you think about this?

ivanpavlina commented 3 years ago

Oooh, breaking changes, yay :)

I didn't try v7 so I didn't notice that they are removing accounting...

I will check it out in the coming days, first I will try to make PR just so the missing accounting feature isn't breaking whole integration.. And then I will check out traffic flow. I really hope it's somewhat similar to accounting :D

tomaae commented 3 years ago

Yea, unfortunately. We already have version in resource>version, in case that would be the best way to disable it quickly.

jwshields commented 3 years ago

If you would like me to do any testing on my devices, I am more than willing to help out with that - I have both an RB4011 & a cAP AC device running the 7.1 beta.

ivanpavlina commented 3 years ago

Sorry for taking so long, work is hectic, I created PR #65 which will check if the version is 7 or up - if it is accounting data won't be pulled from mikrotik.

About traffic-flow, I glanced over it a bit, and I don't think there's a point to implement it in this component. You have to explicitly register which interfaces and addresses you want to monitor and this defeats the whole purpose - accounting was working fine 'out-of-the-box', component only had to know which network is local so it could differentiate traffic. But traffic flow would complicate things. I don't know, maybe you can try to implement this functionality if its possible, I'm overwhelmed with work atm and cant even start looking into it at least until next year :(

jwshields commented 3 years ago

From what I can tell, at least on the Mikrotik side of things, you set the settings on which interfaces to monitor - the targets area is where the data is sent to. I don't believe you've got to specify each target-dest combo to monitor, because that would just be insane. https://wiki.mikrotik.com/wiki/Manual:IP/Traffic_Flow#Targets

jwshields commented 3 years ago

I enabled traffic flow, and had tried to send it to my local Splunk instance, but it didn't parse it properly. But it was definitely capturing flows while enabled, with or without a target An intermediary daemon might be required to receive & handle traffic flow packets for this integration if traffic flow is really wanted/needed

jwshields commented 3 years ago

Accidentally closed, sorry

tomaae commented 3 years ago

ivan seems to be very busy, so I implemented code from his PR bit differently. integration should work on FW>7 now, but I have no way to test it myself.

mrtnbr commented 3 years ago

I am on RouterOS v7.1beta3 and I am having these problems (I think it is the same issue). I switched to master on HACS and reconfigured the Integration. The devices show up, but the entities are unavailable.

Excerpt from the HA log:

2020-12-08 20:58:14 WARNING (SyncWorker_0) [custom_components.mikrotik_router.mikrotikapi] Mikrotik Reconnected to 192.168.0.1 2020-12-08 20:58:14 ERROR (SyncWorker_3) [custom_components.mikrotik_router.mikrotikapi] Mikrotik 192.168.0.1 error while building list for path : unknown 2020-12-08 20:58:14 WARNING (SyncWorker_3) [custom_components.mikrotik_router.mikrotikapi] Mikrotik Reconnected to 192.168.0.1 2020-12-08 20:58:14 ERROR (SyncWorker_3) [custom_components.mikrotik_router.mikrotikapi] Mikrotik 192.168.0.1 error while accounting_snapshot : unknown

tomaae commented 3 years ago

can you debug it?

mrtnbr commented 3 years ago

can you debug it?

I will try tonight, but I'm not really sure how to do it :)

tomaae commented 3 years ago

can you debug it?

I will try tonight, but I'm not really sure how to do it :)

Here you go: https://github.com/tomaae/homeassistant-mikrotik_router#enabling-debug

mrtnbr commented 3 years ago

So, to summarize: RouterOS v7.1beta3, integration version Master/ 5e7ae94 via HACS. Debug log excerpt:

2020-12-09 20:26:48 DEBUG (SyncWorker_14) [custom_components.mikrotik_router.mikrotikapi] API query: /ip/accounting 2020-12-09 20:26:48 ERROR (SyncWorker_14) [custom_components.mikrotik_router.mikrotikapi] Mikrotik 192.168.0.1 error while building list for path : unknown 2020-12-09 20:26:48 DEBUG (SyncWorker_14) [custom_components.mikrotik_router.mikrotik_controller] Working with 23 accounting devices 2020-12-09 20:26:48 WARNING (SyncWorker_14) [custom_components.mikrotik_router.mikrotikapi] Mikrotik Reconnected to 192.168.0.1 2020-12-09 20:26:48 DEBUG (SyncWorker_14) [custom_components.mikrotik_router.mikrotikapi] API query: /ip/accounting 2020-12-09 20:26:48 ERROR (SyncWorker_14) [custom_components.mikrotik_router.mikrotikapi] Mikrotik 192.168.0.1 error while accounting_snapshot : unknown 2020-12-09 20:26:48 DEBUG (MainThread) [custom_components.mikrotik_router.sensor] Updating sensor Mikrotik-system_uptime

tomaae commented 3 years ago

I will need at least everything under api request /system/routerboard There are data related to firmware version.

mrtnbr commented 3 years ago

Interesting. I might have done something wrong in the upgrade process of the router, sorry about that!:

2020-12-10 11:03:44 DEBUG (SyncWorker_3) [custom_components.mikrotik_router.mikrotikapi] API query: /system/routerboard 2020-12-10 11:03:44 DEBUG (SyncWorker_3) [custom_components.mikrotik_router.helper] Processing source [{'routerboard': True, 'board-name': 'hEX PoE', 'model': 'RB960PGS', 'serial-number': '', 'firmware-type': 'qca9550L', 'factory-firmware': '6.44', 'current-firmware': '6.47.7', 'upgrade-firmware': '7.1beta3'}] 2020-12-10 11:03:44 DEBUG (SyncWorker_3) [custom_components.mikrotik_router.helper] Processing entry {'routerboard': True, 'board-name': 'hEX PoE', 'model': 'RB960PGS', 'serial-number': '', 'firmware-type': 'qca9550L', 'factory-firmware': '6.44', 'current-firmware': '6.47.7', 'upgrade-firmware': '7.1beta3'}

image

After router reboot:

2020-12-10 11:25:28 DEBUG (SyncWorker_2) [custom_components.mikrotik_router.mikrotikapi] API query: /system/routerboard 2020-12-10 11:25:28 DEBUG (SyncWorker_2) [custom_components.mikrotik_router.helper] Processing source [{'routerboard': True, 'board-name': 'hEX PoE', 'model': 'RB960PGS', 'serial-number': '', 'firmware-type': 'qca9550L', 'factory-firmware': '6.44', 'current-firmware': '7.1beta3', 'upgrade-firmware': '7.1beta3'}] 2020-12-10 11:25:28 DEBUG (SyncWorker_2) [custom_components.mikrotik_router.helper] Processing entry {'routerboard': True, 'board-name': 'hEX PoE', 'model': 'RB960PGS', 'serial-number': '', 'firmware-type': 'qca9550L', 'factory-firmware': '6.44', 'current-firmware': '7.1beta3', 'upgrade-firmware': '7.1beta3'}

image

Now the accounting error seems to have disappeared, but there is another one with the API query: /system/script/environment

2020-12-10 11:25:28 DEBUG (SyncWorker_2) [custom_components.mikrotik_router.mikrotikapi] API query: /queue/simple 2020-12-10 11:25:28 DEBUG (SyncWorker_1) [custom_components.mikrotik_router.mikrotikapi] API query: /system/script/environment 2020-12-10 11:25:28 ERROR (SyncWorker_1) [custom_components.mikrotik_router.mikrotikapi] Mikrotik 192.168.0.1 error while building list for path : unknown 2020-12-10 11:25:28 INFO (MainThread) [homeassistant.setup] Setting up device_tracker 2020-12-10 11:25:28 INFO (MainThread) [homeassistant.components.binary_sensor] Setting up binary_sensor.mikrotik_router 2020-12-10 11:25:28 DEBUG (MainThread) [custom_components.mikrotik_router.binary_sensor] Updating binary_sensor Mikrotik-system_fwupdate 2020-12-10 11:25:28 INFO (MainThread) [homeassistant.setup] Setup of domain device_tracker took 0.0 seconds 2020-12-10 11:25:28 INFO (MainThread) [homeassistant.components.device_tracker] Setting up device_tracker.mikrotik_router

tomaae commented 3 years ago

Environment too? They sure do some large changes in that upcoming release. I need to get a device I can install routeros 7 on, so I can test everything and implement new methods.

mrtnbr commented 3 years ago

Environment too? They sure do some large changes in that upcoming release. I need to get a device I can install routeros 7 on, so I can test everything and implement new methods.

I tried to get the /system/script/environment output by using a python script to do API calls, but could not get it to authenticate on the router unfortunately. I'll try again.

jwshields commented 3 years ago

If you could give further instructions on how to get these fixes/changes and what to debug, running on hassio, I can test them on my router & cap and provide feedback/changes.

image image

tomaae commented 3 years ago

this should not affect you on 6.45. its only affects 7+ you can get these changes by installing master. you can do that in hacs with version selector. but I need to get a device to test and adapt to 7+ myself. best with wifi, so there is full feature set. I will look for something cheap 2nd hand, like cAP lite. Hopefully I can get one cheap.

jwshields commented 3 years ago

I am not on 6.45. My rb4011 is running 7.1beta2 and my cap ac is running 7.1beta3. I am not sure why the "current firmware" is misreporting. Could be a Winbox bug?

tomaae commented 3 years ago

could be, check what this reports: /system routerboard print

jwshields commented 3 years ago

RB4011 image

CAP AC image

tomaae commented 3 years ago

oh right, you didnt upgrade firmware, just OS. press upgrade button and wait few seconds until it shows done. reboot afterwards. I will need to add a check for this too.

jwshields commented 3 years ago

I am performing the upgrades now, but I'm sorry, but I don't see the issue here, what has that got to do with this overall? The devices are running the 7.1 beta OS, but the bootloader is the old version? What effect is that having on the hassio addon?

tomaae commented 3 years ago

it uses that to detect version. I keep bootloader up to date, so didnt noticed it was like that.

tomaae commented 3 years ago

I will get 2nd hand cAP lite next week. That will make things much easier.

tomaae commented 3 years ago

I have created new issue for 7+ support, we will move all discussion which is not related to accounting and traffic flow there to keep this on topic. #73

tomaae commented 3 years ago

I have fixed version detection, so this could work. Unless there are more breaking changes of course.

tomaae commented 3 years ago

integration now works on my device with 7.1beta3 I didnt had any issues with environment. if you see them again, debug it and report the issue in #73

jwshields commented 3 years ago

Still seeing this issue where everything is unavailable. (As seen in the screenshots in my first message) Completely reinstalled the integration, used all defaults for the first basic section.

RouterOS: 7.1beta2 Mikrotik Router integration version: 1.7.2 hassio: core-2021.6.6

Closing issue, this integration is no longer relevant to my interests

tomaae commented 3 years ago

Well, you need to cooperate if you want an issue specific to you fixed. Not much I can do without debugs.

jwshields commented 3 years ago

I provided a debug log in the original post. Does the integration rely on traffic flow/accounting for device trackers? The misdirection of all the comments here, regarding accounting, version 7, os version vs firmware... weren't addressing the issue of entities becoming unavailable after each poll.

The debug logs that I saw shortly before I closed this issue were more or less the same as the Gist I linked. It would update whatever entities and they would become unavailable immediately after.

tomaae commented 3 years ago

There was no misdirection, that is what was happening in debug log you originally provided and that issue was fixed. Then that, you were facing issue with firmware version detection being based on bootloader version, which was fixed too. After that point you no longer replied so of course I assumed everything was fixed and left this open until Ivan has time to take a look into new implementation for traffic flow. Integration does not rely on traffic flow, it checks if you have it enabled which used to report an error on 7+ as they removed the feature.

Debug logs will look similar all the time of course, but you certainly should not have same issue as that was fixed. I believe you dont see entities, but there is different problem. I can fix it, but I will need debugs logs using current integration version.

Since I have same type of router as my main driver and ended up buying same ap you have as part of this issue, you most likely have some special setup. Can be a small thing, but I need to be able to see what is happening.