search

openhab / openhab-addons

Add-ons for openHAB
https://www.openhab.org/
Eclipse Public License 2.0
1.9k stars 3.59k forks source link

[Netatmo] Polling leads to missing values in some channels - Smart Video Doorbell #13002

Closed HaKuNaCH closed 2 months ago

HaKuNaCH commented 2 years ago

Issue came-up during the webhook investigations in : https://github.com/openhab/openhab-addons/pull/12972#issuecomment-1164042787

Expected Behavior

Periodical polling should deliver the values from /getevents

Current Behavior

Polling does update the following channels with NULL:

sub-event#snapshot
sub-event#snapshot-url
sub-event#event
sub-event#event-url
sub-event#message

Possible Solution

Values must be taken from /getevents call. Example (I left 1 event in, but all the follwowing events also contain the same attributes and all with values):

{
    body: {
        home: {
            id: "xxxxxxxxxxxxxxxxxxxxxxxx",
            events: [{
                id: "xxxxxxxxxxxxxxxxxxxxxxxx",
                type: "outdoor",
                time: 1655917778,
                module_id: "xx:xx:xx:xx:xx:xx",
                video_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
                video_status: "available",
                subevents: [{
                    id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
                    type: "human",
                    time: 1655917778,
                    verified: true,
                    offset: 1,
                    snapshot: {
                        url: "https://netatmocameraimage.blob.core.windows.net/production/xxxxxxxxxxxxxxxxxxx",

                    },
                    vignette: {
                        url: "https://netatmocameraimage.blob.core.windows.net/production/xxxxxxxxxxxxxxxxxxx",

                    },
                    message: "Person erfasst"
                }]
            },

            .. around 40 entries, all with the same attributtes and values are present in all ..

            ]
        }
    },
    status: "ok",
    time_exec: 0.042575836181640625,
    time_server: 1655967343
}

If required, I can also post examples from: /homedata /homestatus

According to the documentation, all other GET API's are deprecated https://dev.netatmo.com/apidocumentation/security#documentation

Steps to Reproduce (for Bugs)

  1. Install/configure netatmo binding incl. webhook
  2. add all available channels
  3. add item to the channels
  4. check the values after polling
lolodomo commented 2 years ago

@clinique : any news?

lolodomo commented 2 years ago

Reading again this issue, I understand that something is not correctly handled when an event contains sub-event(s). The sub-event(s) would be ignored. Do you agree @clinique ? The problem is that we don't have at channel level this notion of event and sub-event. Should we merge the event with its sub-event ? What to do if an event contains several sub-events ? Should we trigger several event updates on channels level ?

openhab-bot commented 1 year ago

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-get-netatmo-doorbell-working/145303/6

clinique commented 7 months ago

Reading again this issue, I understand that something is not correctly handled when an event contains sub-event(s). The sub-event(s) would be ignored. Do you agree @clinique ? The problem is that we don't have at channel level this notion of event and sub-event. Should we merge the event with its sub-event ? What to do if an event contains several sub-events ? Should we trigger several event updates on channels level ?

I adressed this one while working on the resolution of #15883, the PR is this one : https://github.com/openhab/openhab-addons/pull/16681

SubEvents are now handled. One event can have multiple subevent and each sub will be handled sequentially.

lolodomo commented 7 months ago

@clinique : is this issue now fixed with the last merges ?

clinique commented 7 months ago

From my standpoint and understanding of the issue, yes.

lolodomo commented 7 months ago

Thank you @clinique @HaKuNaCH : please reopen if your issue is still not fully resolved after all the recent changes.

HaKuNaCH commented 6 months ago

Unfortunately, I cannot reopen the issue.

The issue is not fixed. The following channels get the state, but the state gets overwritten immediately with NULL. At least on OH 4.2.0.M3.

Affected channels: sub-event#snapshot sub-event#snapshot-url sub-event#vignette sub-event#vignette-url last-event#video-status last-event#local-video-url last-event#vpn-video-url

Example for last-event#local-video-url:

2024-05-21 09:53:23.509 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoLocalURL' changed from NULL to http://10.0.1.37/5695f56d15bb3cdad9b7930597a26f91/vod/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/index.m3u8
2024-05-21 09:53:23.510 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoLocalURL' changed from http://10.0.1.37/5695f56d15bb3cdad9b7930597a26f91/vod/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/index.m3u8 to NULL
2024-05-21 09:56:24.769 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoLocalURL' changed from NULL to http://10.0.1.37/5695f56d15bb3cdad9b7930597a26f91/vod/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/index.m3u8
2024-05-21 09:56:24.770 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoLocalURL' changed from http://10.0.1.37/5695f56d15bb3cdad9b7930597a26f91/vod/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/index.m3u8 to NULL
2024-05-21 09:59:25.959 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoLocalURL' changed from NULL to http://10.0.1.37/5695f56d15bb3cdad9b7930597a26f91/vod/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/index.m3u8
2024-05-21 09:59:25.960 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoLocalURL' changed from http://10.0.1.37/5695f56d15bb3cdad9b7930597a26f91/vod/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/index.m3u8 to NULL

Could you please reopen the issue. If you need more logs, please contact me. Thanks

HaKuNaCH commented 4 months ago

Currently, I'm using a workaround to have the lastest state of the items, by adding the state if not NULL to a temporary item via a jython rule.

@rule("Netatmo update temorary items", description="Updates the temporary netatmo items if not NULL", tags=["netatmo"])
@rule("Netatmo update temorary items", description="Updates the temporary netatmo items if not NULL", tags=["netatmo"])
@when("Item DoorbellEventSnapshot changed")
@when("Item DoorbellEventSnapshotUrl changed")
@when("Item DoorbellVignette changed")
@when("Item DoorbellVignetteUrl changed")
@when("Item DoorbellLastVideoStatus changed")
@when("Item DoorbellLastVideoLocalURL changed")
@when("Item DoorbellLastVideoVpnURL changed")
def netatmoUpdateTempItems(event):
    #log.warn("Rule: Netatmo netatmoUpdateTempItems started.")

    if unicode(event.itemState) != "NULL":
        events.postUpdate(unicode(event.itemName) + u"Temp", unicode(event.itemState.toFullString()))
        #log.warn("DEBUG: ITEM=" + unicode(event.itemName) + u"Temp updated.")

    #log.warn("Rule: Netatmo netatmoUpdateTempItems ended.")
lsiepel commented 3 months ago

@clinique are you able to comment? You seem to be the expert on this binding ;-)

clinique commented 2 months ago

I'm coming back on this issue.

clinique commented 2 months ago

Hello @HaKuNaCH . Could you please try this patched version and give me your feed-back. This involves webhook and is difficult to reproduce for me, I'll need your help.

HaKuNaCH commented 2 months ago

Thanks @clinique I just updated to 4.3.0.M1 and installed your patched version of the Netatmo binding. I'll observe the events in the next couple of days and will provide feedback here.

HaKuNaCH commented 2 months ago

Here are my findings.

These are still NOK -> they are updated correctly on event, but finally got updated with NULL again zcat -f events.log* | grep "'DoorbellEventSnapshot'" | sort

2024-09-05 12:44:00.310 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from NULL to raw type (image/jpeg): 336553 bytes
2024-09-05 12:46:13.386 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from raw type (image/jpeg): 336553 bytes to NULL
2024-09-05 12:46:13.442 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from NULL to raw type (image/jpeg): 336553 bytes
2024-09-05 12:49:14.474 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from raw type (image/jpeg): 336553 bytes to NULL
2024-09-05 12:49:14.543 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from NULL to raw type (image/jpeg): 336553 bytes
2024-09-05 12:52:15.510 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from raw type (image/jpeg): 336553 bytes to NULL
2024-09-06 16:45:31.497 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from NULL to raw type (image/jpeg): 173289 bytes
2024-09-06 16:45:59.592 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from raw type (image/jpeg): 173289 bytes to NULL
2024-09-06 16:45:59.611 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from NULL to raw type (image/jpeg): 173289 bytes
2024-09-06 16:49:00.885 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from raw type (image/jpeg): 173289 bytes to NULL
2024-09-06 16:49:00.907 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from NULL to raw type (image/jpeg): 173289 bytes
2024-09-06 16:52:02.809 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from raw type (image/jpeg): 173289 bytes to NULL

zcat -f events.log* | grep "'DoorbellVignette'" | sort

2024-09-05 12:44:00.316 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from NULL to raw type (image/jpeg): 71159 bytes
2024-09-05 12:46:13.388 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from raw type (image/jpeg): 71159 bytes to NULL
2024-09-05 12:46:13.450 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from NULL to raw type (image/jpeg): 71159 bytes
2024-09-05 12:49:14.476 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from raw type (image/jpeg): 71159 bytes to NULL
2024-09-05 12:49:14.551 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from NULL to raw type (image/jpeg): 71159 bytes
2024-09-05 12:52:15.511 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from raw type (image/jpeg): 71159 bytes to NULL
2024-09-06 16:45:31.521 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from NULL to raw type (image/jpeg): 16881 bytes
2024-09-06 16:45:59.592 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from raw type (image/jpeg): 16881 bytes to NULL
2024-09-06 16:45:59.627 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from NULL to raw type (image/jpeg): 16881 bytes
2024-09-06 16:49:00.885 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from raw type (image/jpeg): 16881 bytes to NULL
2024-09-06 16:49:00.921 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from NULL to raw type (image/jpeg): 16881 bytes
2024-09-06 16:52:02.810 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from raw type (image/jpeg): 16881 bytes to NULL

These are "partially" OK -> but periodically updated to NULL, afterwards they are updated again with the last state zcat -f events.log* | grep "'DoorbellEventSnapshotUrl'" | sort

2024-09-06 16:45:31.498 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-06 16:45:59.592 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 16:45:59.611 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-06 16:49:00.885 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 16:49:00.908 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-06 16:52:02.809 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 16:52:02.873 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-06 16:55:04.136 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 16:55:04.141 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-06 16:58:05.528 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 16:58:05.537 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...

zcat -f events.log* | grep "'DoorbellVignetteUrl'" | sort

2024-09-06 16:49:00.886 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 16:49:00.921 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-06 16:52:02.810 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 16:52:02.891 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-06 16:55:04.136 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 16:55:04.158 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-06 16:58:05.528 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 16:58:05.621 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-06 17:01:07.757 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to NULL
2024-09-06 17:01:07.781 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from NULL to https://netatmocameraimage.blob.core.windows.net/production/...

These are OK -> no update to NULL zcat -f events.log* | grep "'DoorbellLastVideoStatus'" | sort

2024-09-04 20:03:10.293 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoStatus' changed from NULL to AVAILABLE

zcat -f events.log* | grep "'DoorbellLastVideoLocalURL'" | sort

2024-09-05 12:46:13.379 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoLocalURL' changed from http://10.0.1.37/.../index.m3u8 to http://10.0.1.37/.../index.m3u8
2024-09-06 16:45:59.586 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoLocalURL' changed from http://.../index.m3u8 to http://.../index.m3u8

zcat -f events.log* | grep "'DoorbellLastVideoVpnURL'" | sort

2024-09-05 15:32:27.391 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoVpnURL' changed from https://prodvpn-eu-13.netatmo.net/restricted/.../index.m3u8 to https://prodvpn-eu-13.netatmo.net/restricted/.../index.m3u8
2024-09-06 16:45:59.586 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoVpnURL' changed from https://prodvpn-eu-13.netatmo.net/restricted/.../index.m3u8 to https://prodvpn-eu-1.netatmo.net/restricted/.../index.m3u8
clinique commented 2 months ago

Thanks for your detailed analysis @HaKuNaCH , it is very helpfull. So here's a second corrective version. Let me know. BR

HaKuNaCH commented 2 months ago

Here are my findings.

These are still NOK -> they are updated correctly on event, but finally got updated with NULL again zcat -f events.log* | grep "'DoorbellEventSnapshot'" | sort

2024-09-09 09:19:30.962 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from NULL to raw type (image/jpeg): 177547 bytes
2024-09-09 09:26:40.807 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshot' changed from raw type (image/jpeg): 177547 bytes to NULL

zcat -f events.log* | grep "'DoorbellVignette'" | sort

2024-09-09 09:18:53.164 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from NULL to raw type (image/jpeg): 16559 bytes
2024-09-09 09:26:40.820 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignette' changed from raw type (image/jpeg): 16559 bytes to NULL

These are now OK -> no longer update to NULL zcat -f events.log* | grep "'DoorbellEventSnapshotUrl'" | sort

2024-09-09 09:18:53.141 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-09 09:19:30.962 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from https://netatmocameraimage.blob.core.windows.net/... to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-09 09:20:38.032 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellEventSnapshotUrl' changed from https://netatmocameraimage.blob.core.windows.net/... to https://netatmocameraimage.blob.core.windows.net/production/...

zcat -f events.log* | grep "'DoorbellVignetteUrl'" | sort

2024-09-09 09:18:53.164 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-09 09:19:30.985 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/... to https://netatmocameraimage.blob.core.windows.net/production/...
2024-09-09 09:20:38.046 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellVignetteUrl' changed from https://netatmocameraimage.blob.core.windows.net/production/6... to https://netatmocameraimage.blob.core.windows.net/production/...

These are still OK -> no update to NULL zcat -f events.log* | grep "'DoorbellLastVideoStatus'" | sort

2024-09-09 09:20:37.972 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoStatus' changed from RECORDING to AVAILABLE

zcat -f events.log* | grep "'DoorbellLastVideoLocalURL'" | sort

2024-09-09 09:20:37.972 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoLocalURL' changed from http://10.0.1.37/.../index.m3u8 to http://10.0.1.37/.../index.m3u8

zcat -f events.log* | grep "'DoorbellLastVideoVpnURL'" | sort

2024-09-09 09:32:44.655 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'DoorbellLastVideoVpnURL' changed from https://prodvpn-eu-13.netatmo.net/restricted/.../index.m3u8 to https://prodvpn-eu-13.netatmo.net/restricted/.../index.m3u8
clinique commented 2 months ago

@HaKuNaCH , we're progressing. I will try to provide you with yet another patch soon.

clinique commented 2 months ago

I think I catched the reason why Vignette and Snapshot may not be present, while their url are : https://github.com/openhab/openhab-addons/issues/14838 : they expire after five minutes.

HaKuNaCH commented 2 months ago

Indeed, then it looks like the issue is resolved with your recent adjustments. Thx a lot for your work !

clinique commented 2 months ago

Indeed, then it looks like the issue is resolved with your recent adjustments. Thx a lot for your work !

I'll update it, taking in account the expires_at info. You'll have another patch soon.

clinique commented 2 months ago

Here's a new patch

HaKuNaCH commented 2 months ago

Tested. Now also "EventSnapshot" and "Vignette" keep the last state.