Fortinet Integration has stopped working with FortiOS v7.4.3

SirSheik commented 6 months ago

The problem

Hello. After updating the version of FortiOS from version 7.2 to the latest version available today (7.4.3) has stopped working the integration of home assistant with this platform.

What version of Home Assistant Core has the issue?

core-2024.4.2

What was the last working version of Home Assistant Core?

core-2024.4.2

What type of installation are you running?

Home Assistant OS

Integration causing the issue

fortios

Link to integration documentation on our website

https://www.home-assistant.io/integrations/fortios/

Diagnostics information

It is not possible to download diagnostic information because the integration has not yet been migrated to GUI.

Example YAML snippet

device_tracker:
  - platform: fortios
    host: {FW IP}
    token: !secret fortios_token
    verify_ssl: false
    new_device_defaults:
      track_new_devices: false

Anything in the logs that might be useful for us?

2024-04-09 16:34:04.211 WARNING (SyncWorker_6) [fortiosapi] in formatresponse res.content does not exist, should not occur
2024-04-09 16:34:04.211 ERROR (SyncWorker_6) [homeassistant.components.fortios.device_tracker] Failed to login to FortiOS API: Not logged on a session, please login.
2024-04-09 16:34:04.213 ERROR (MainThread) [homeassistant.components.device_tracker] Error setting up platform legacy fortios

Additional information

Attached is a screenshot of the user's permissions in Fortinet.

These permissions were sufficient when the integration was working. However try to generate a new API Key and give read permissions to the user in all options. But it did not work

home-assistant[bot] commented 6 months ago

Hey there @kimfrellsen, mind taking a look at this issue as it has been labeled with an integration (fortios) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `fortios` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign fortios` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

fortios documentation fortios source _{^{(message by IssueLinks)}}

kimfrellsen commented 6 months ago

Hi, It looks like an authentication issue. Have you tried to give the api user more permissions? I have it working with FOS version 7.4.3. It also uses the system api to detect FOS version. /Kim

On Tue, Apr 9, 2024 at 4:54 PM home-assistant[bot] @.***> wrote:

Hey there @kimfrellsen https://github.com/kimfrellsen, mind taking a look at this issue as it has been labeled with an integration (fortios) you are listed as a code owner https://github.com/home-assistant/core/blob/dev/CODEOWNERS#L459 for? Thanks! Code owner commands

Code owners of fortios can trigger bot actions by commenting:

@home-assistant close Closes the issue.

@home-assistant rename Awesome new title Renames the issue.

@home-assistant reopen Reopen the issue.

@home-assistant unassign fortios Removes the current integration label and assignees on the issue, add the integration domain after the command.

@home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.

@home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)

fortios documentation https://www.home-assistant.io/integrations/fortios fortios source https://github.com/home-assistant/core/tree/dev/homeassistant/components/fortios (message by IssueLinks)

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/115286#issuecomment-2045386652, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7LXYYSNMMZ7QNXUNG3BMDY4P6LNAVCNFSM6AAAAABF6XXZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBVGM4DMNRVGI . You are receiving this because you were mentioned.Message ID: @.***>

SirSheik commented 6 months ago

Hi, It looks like an authentication issue. Have you tried to give the api user more permissions? I have it working with FOS version 7.4.3. It also uses the system api to detect FOS version. /Kim … On Tue, Apr 9, 2024 at 4:54 PM home-assistant[bot] @.> wrote: Hey there @kimfrellsen https://github.com/kimfrellsen, mind taking a look at this issue as it has been labeled with an integration (fortios) you are listed as a code owner https://github.com/home-assistant/core/blob/dev/CODEOWNERS#L459 for? Thanks! Code owner commands Code owners of fortios can trigger bot actions by commenting: - @home-assistant close Closes the issue. - @home-assistant rename Awesome new title Renames the issue. - @home-assistant reopen Reopen the issue. - @home-assistant unassign fortios Removes the current integration label and assignees on the issue, add the integration domain after the command. - @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue. (message by CodeOwnersMention) ------------------------------ fortios documentation https://www.home-assistant.io/integrations/fortios fortios source https://github.com/home-assistant/core/tree/dev/homeassistant/components/fortios (message by IssueLinks) — Reply to this email directly, view it on GitHub <#115286 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7LXYYSNMMZ7QNXUNG3BMDY4P6LNAVCNFSM6AAAAABF6XXZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBVGM4DMNRVGI . You are receiving this because you were mentioned.Message ID: @.>

Hi @kimfrellsen First of all thank you for your quick response. I have just granted all read and write permissions to the specific group I have created for the home assistant user, but the problem remains.

Could you share some screenshots of your configuration in case I have some misconfigured option that is needed in this new version of FortiOS? Thank you very much for your time.

NBD99 commented 6 months ago

I can confirm i have the same issue when upgrading from FortiOS 7.2.7 to 7.4.3.

However it seems like the issue could be in FortiOS since almost none of my devices goes offline in the Fortigate. I will do some more digging.

In my case i don´t have any error messages in HA, the entities just never change state they are always home no matter what.

NBD99 commented 6 months ago

As i suspected something seems to bee broken in FortiOS 7.4.3 when it comes to device tracking.

For example the last seen flag keeps updating for my phone even though its not connected to the network. This goes for both the CLI and GUI, that maybe explains why devices never seem to go offline once they have once connected regardless if they disconnect.

SirSheik commented 6 months ago

Hi @kimfrellsen I don't know if this helps, but I've been debugging Home Assistant's API calls to Fortigate and I've detected that it shows the following error.

The debugging from Fortigate was done with the following commands

$ diagnose debug enable
$ diagnose debug application httpsd -1
$ diagnose debug cli 8

I also tried deleting all the entities in Fortigate so that it would re-identify them using the following command and at least the first time I got Home Assistant to refresh the correct state of the entities, but this only worked the first time it detects the device, after that Home Assistant does not recover the updated state.

diag user device clear
diag user device list

Could you tell me how is the API call that home assistant makes to debug on my system and try to help with this problem? Thx

kimfrellsen commented 6 months ago

Hi SirSheik, Thank for the help. Especially the filter issue.

I too think there is a bug in device detection on FOS 7.4.x. I'll check bug fixes on 7.4.x and if needed open a bug request.

I have also looked at your initial issue "2024-04-09 16:34:04.211 WARNING (SyncWorker_6) [fortiosapi] in formatresponse res.content does not exist, should not occur 2024-04-09 16:34:04.211 ERROR (SyncWorker_6) [homeassistant.components.fortios.device_tracker] Failed to login to FortiOS API: Not logged on a session, please login. 2024-04-09 16:34:04.213 ERROR (MainThread) [homeassistant.components.device_tracker] Error setting up platform legacy fortios"

I can reproduce this using api calls through the external component FortiOSAPI. This python component is a bit outdated and no longer actively maintained. It is a python library dependency issue. So I'll update the HASS integration to use python requests directly to FortiGate instead.

Time to update the integration to get rid of YAML and FortiOSAPI :) /Kim

SirSheik commented 6 months ago

Hi SirSheik, Thank for the help. Especially the filter issue.

I too think there is a bug in device detection on FOS 7.4.x. I'll check bug fixes on 7.4.x and if needed open a bug request.

I have also looked at your initial issue "2024-04-09 16:34:04.211 WARNING (SyncWorker_6) [fortiosapi] in formatresponse res.content does not exist, should not occur 2024-04-09 16:34:04.211 ERROR (SyncWorker_6) [homeassistant.components.fortios.device_tracker] Failed to login to FortiOS API: Not logged on a session, please login. 2024-04-09 16:34:04.213 ERROR (MainThread) [homeassistant.components.device_tracker] Error setting up platform legacy fortios"

I can reproduce this using api calls through the external component FortiOSAPI. This python component is a bit outdated and no longer actively maintained. It is a python library dependency issue. So I'll update the HASS integration to use python requests directly to FortiGate instead.

Time to update the integration to get rid of YAML and FortiOSAPI :) /Kim

Thank you Kim. If I can help in any way you let me know. If I get some time I will try to investigate a little more to see if I can help in solving the problem. Keep in touch in this topic for possible progress. Thanks for your time.

kimfrellsen commented 6 months ago

Hi SirSheik,

I have successfully refactored the integration to get remove dependencies to the python library FortiOSAPI and added support for vdoms.

I cannot push it as Home-Assistant (with a very valid reason) does not accept pull request for integrations which rely on yaml configuration, so I am also rewriting the my integration to support config_flow. When I have it working I'll let you know, maybe you can help to test it.

/Kim

On Tue, Apr 23, 2024 at 4:37 PM SirSheik @.***> wrote:

Hi SirSheik, Thank for the help. Especially the filter issue.

I too think there is a bug in device detection on FOS 7.4.x. I'll check bug fixes on 7.4.x and if needed open a bug request.

I have also looked at your initial issue "2024-04-09 16:34:04.211 WARNING (SyncWorker_6) [fortiosapi] in formatresponse res.content does not exist, should not occur 2024-04-09 16:34:04.211 ERROR (SyncWorker_6) [homeassistant.components.fortios.device_tracker] Failed to login to FortiOS API: Not logged on a session, please login. 2024-04-09 16:34:04.213 ERROR (MainThread) [homeassistant.components.device_tracker] Error setting up platform legacy fortios"

I can reproduce this using api calls through the external component FortiOSAPI. This python component is a bit outdated and no longer actively maintained. It is a python library dependency issue. So I'll update the HASS integration to use python requests directly to FortiGate instead.

Time to update the integration to get rid of YAML and FortiOSAPI :) /Kim

Thank you Kim. If I can help in any way you let me know. If I get some time I will try to investigate a little more to see if I can help in solving the problem. Keep in touch in this topic for possible progress. Thanks for your time.

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/115286#issuecomment-2072504328, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7LXY7LW452P7VFXUXKH5LY6ZW3BAVCNFSM6AAAAABF6XXZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZSGUYDIMZSHA . You are receiving this because you were mentioned.Message ID: @.***>

Phillip411 commented 6 months ago

I will also be glad to help test as I am setting up the integration for the first time and it is failing for me as well.

kimfrellsen commented 6 months ago

Thanks guys, very nice. I found a number of issues. One in fortiosapi, which I have now refactored to get rid of in the component. Next one issue in the component which is fixed. I also found an oddity in the API response from FortiOS, this is reported so let's see how that goes. Lastely I noticed the base class "DeviceScanner" which I am inheriting in the component is know put in to legacy state, so I have to refactor the whole component to be async and use config_flow to be inline with hass guidelines. This will take some time, but in the end it will result in a much better component. Thanks for finding the issues. I'll push the first minor changes to my fork at: https://github.com/kimfrellsen/core/tree/dev/homeassistant/components/fortios You are welcome to test it. Of couse it does not fix the oddity with the FOS API response. You can install it as a custom component. custom components takes precedence over regular components. I'll keep you posted. Would be great with some testing when rewriting is finished. Have a great day. /Kim

Phillip411 commented 6 months ago

Thank you Kim !!! This sounds great.

I will keep my eyes peeled for updates to your code. For reference my fortios version is v7.2.7 build1577

On Fri, Apr 26, 2024 at 7:00 AM Kim Frellsen @.***> wrote:

Thanks guys, very nice. I found a number of issues. One in fortiosapi, which I have now refactored to get rid of in the component. Next one issue in the component which is fixed. I also found an oddity in the API response from FortiOS, this is reported so let's see how that goes. Lastely I noticed the base class "DeviceScanner" which I am inheriting in the component is know put in to legacy state, so I have to refactor the whole component to be async and use config_flow to be inline with hass guidelines. This will take some time, but in the end it will result in a much better component. Thanks for finding the issues. I'll push the first minor changes to my fork at: https://github.com/kimfrellsen/core/tree/dev/homeassistant/components/fortios You are welcome to test it. Of couse it does not fix the oddity with the FOS API response. You can install it as a custom component. custom components takes precedence over regular components. I'll keep you posted. Would be great with some testing when rewriting is finished. Have a great day. /Kim

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/115286#issuecomment-2079248899, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTELQ52TP3PE5J5F4PDO23Y7I6XPAVCNFSM6AAAAABF6XXZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZZGI2DQOBZHE . You are receiving this because you commented.Message ID: @.***>

Phillip411 commented 6 months ago

One quick update. I updated to firmware version v7.2.8 and the integration started working (eithout the need of using custom_components). I hope this information is helpful. Thanks again !!!

SirSheik commented 6 months ago

One quick update. I updated to firmware version v7.2.8 and the integration started working (eithout the need of using custom_components). I hope this information is helpful. Thanks again !!!

Hi. The initial problem was detected in FortiOS v7.4.3. The version you mention is older. In FortiOS v7.2.X version there was no such problem as I initially mentioned. Please try not to mix up errors or we will go back to Kim crazy.

kimfrellsen commented 6 months ago

Hi. I agree the issue is not seen on FOS v.7.2.x, it is only seen on FOS v.7.4.x. Don't worry I am on top of it. Keep you posted. /Kim

SirSheik commented 6 months ago

Thank you very much @kimfrellsen I have already installed your integration as a custom component. Sorry, should I notice any improvements or anything you want me to test? The configuration parameters in the configuration.yaml file are the same, right? Best regards

kimfrellsen commented 6 months ago

@SirSheik

I have put an update here on pastebin: https://pastebin.com/c6JJpPPe

(yes I know pastbin is probably not the safest place, but it is there now).

If you have installed the fortios component as a custom component you can replace the code with the one shared on pastebin, I can also email it to you. it basically fixes a little issue and removes the use of fortiosapi. Naturally it does not fix the issue with the API response in FortiOS v.7.4.x

goncal commented 4 months ago

Hi, I'm running FortiOS v.7.4.4 and also experiencing the problem. If you need me to do any troubleshooting just let me know!

Shall I get the update you mentioned on April 30 (the pastebin URL no longer works), or am I OK using the code at https://github.com/kimfrellsen/core/tree/dev/homeassistant/components/fortios ? By the way, I guess that I just need to drop that code into the custom_components folder and restart HA, right? Best

kimfrellsen commented 3 months ago

Hi,

The root course is FortiOS not updating the online status correctly. I can share the source code I pasted to pastbin with you, it solves the problem with excessive alerts in hass, but it does not solve the problem in fortios.

/Kim

On Tue, Jul 2, 2024 at 5:52 PM goncal @.***> wrote:

Hi, I'm running FortiOS v.7.4.4 and also experiencing the problem. If you need me to do any troubleshooting just let me know!

Shall I get the update you mentioned on April 30 (the pastebin URL no longer works), or am I OK using the code at https://github.com/kimfrellsen/core/tree/dev/homeassistant/components/fortios ? By the way, I guess that I just need to drop that code into the custom_components folder and restart HA, right? Best

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/115286#issuecomment-2203628655, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7LXY4OIY23KDI7OMDAQB3ZKLEC7AVCNFSM6AAAAABF6XXZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBTGYZDQNRVGU . You are receiving this because you were mentioned.Message ID: @.***>

haffi78 commented 3 months ago

And how do I use the Vdom Feature on the custom component, p.s. Thanks for all the help and work ! 👍

Can you also share code with me so I can test :)

kimfrellsen commented 3 months ago

happy to help.

Sorry about the mess.

The problem is to create new configuration parameter require changes in home-assistant core. an easy but awful workaround is to set your custom vdom by changing the code like this in line 48 "vdom": "root", --> "vdom": "your_custom_vdom_name",

/Kim

On Wed, Jul 10, 2024 at 3:15 PM Hafþór Hilmarsson @.***> wrote:

And how do I use the Vdom Feature on the custom component, p.s. Thanks for all the help and work ! 👍

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/115286#issuecomment-2220483990, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7LXY5TVLJGD6K2IMQTFY3ZLUXX3AVCNFSM6AAAAABF6XXZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQGQ4DGOJZGA . You are receiving this because you were mentioned.Message ID: @.***>

kimfrellsen commented 3 months ago

and sorry I did not had time to upgrade the component to a zeroconf component. I am not really good at async programming which is required for a new custom component.

On Wed, Jul 10, 2024 at 3:37 PM Kim Frellsen @.***> wrote:

happy to help.

Sorry about the mess.

The problem is to create new configuration parameter require changes in home-assistant core. an easy but awful workaround is to set your custom vdom by changing the code like this in line 48 "vdom": "root", --> "vdom": "your_custom_vdom_name",

/Kim

On Wed, Jul 10, 2024 at 3:15 PM Hafþór Hilmarsson < @.***> wrote:

And how do I use the Vdom Feature on the custom component, p.s. Thanks for all the help and work ! 👍

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/115286#issuecomment-2220483990, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7LXY5TVLJGD6K2IMQTFY3ZLUXX3AVCNFSM6AAAAABF6XXZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQGQ4DGOJZGA . You are receiving this because you were mentioned.Message ID: @.***>

haffi78 commented 3 months ago

Do you have an url for the changed code the pastebin above is expired, and latest on Git is since aprl :)

migube commented 3 months ago

hi guys, started with fortigate (FortiAP & Fortiswitch) all on latest releases, and indeed same issue for FortiAP, presence= home

For Fortiswitch it doesnt even look that I see the MAC's ? not sure if the new way of working would fix both, but Wifi's more important atm :) tx

haffi78 commented 3 weeks ago

happy to help. Sorry about the mess. The problem is to create new configuration parameter require changes in home-assistant core. an easy but awful workaround is to set your custom vdom by changing the code like this in line 48 "vdom": "root", --> "vdom": "your_custom_vdom_name", /Kim … On Wed, Jul 10, 2024 at 3:15 PM Hafþór Hilmarsson @.> wrote: And how do I use the Vdom Feature on the custom component, p.s. Thanks for all the help and work ! 👍 — Reply to this email directly, view it on GitHub <#115286 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7LXY5TVLJGD6K2IMQTFY3ZLUXX3AVCNFSM6AAAAABF6XXZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQGQ4DGOJZGA . You are receiving this because you were mentioned.Message ID: @.>

Is this still the same issue ? and am I understanding right ? I need to change the code ? So to get vdom in I would need to make a custom component and change the code there ?

goncal commented 3 weeks ago

By the way, just in case it might help... After experiencing some memory exhaustion issues, I opened a case with fortigate and was told that the recommended branch for small fortigates (i.e. those with 2GB of memory like mine) is the 7.2.x. I downgraded to 7.2.x and the integration with HA is now working perfectly (and have not had any more memory exhaustion issues).

kimfrellsen commented 2 weeks ago

Hi, Yes you are correct. I can send you the custom component code to add VDOM support. Correct the recommended release is currently 7.2, see https://community.fortinet.com/t5/FortiGate/Technical-Tip-Recommended-Release-for-FortiOS/ta-p/227178 /Kim

home-assistant / core