search

henry-spanka / node-red-contrib-hikvision

Hikvision API for Node Red
MIT License
10 stars 1 forks source link

Connection to camera is closed after a while #8

Closed ozett closed 4 years ago

ozett commented 4 years ago

does this come from your latest code change? if yes: could it be a little bit more "telling"? ๐Ÿ˜‰

image

henry-spanka commented 4 years ago

I only added the โ€žconnection closed!โ€œ message. I donโ€™t think that the message is logged by the hikvision plugin. If you say that the camera is wired I canโ€™t really think of any issue unless hikvision changed their protocol in the last firmware that lets the connection timeout after a while. Can you try to ping the camera and check if any packets are lost when nodered loses the connection?

ozett commented 4 years ago

i can reach the cameras in the ui, there are also connected to a hikvision nvr, which is recording. i can think of a bandwith-issue, but i dont think, because there are "normally" reachable. only the alarm-polling via node-red seems affected... maby polling intervall is too short?

i disabled almost most nodes, to see where the time-out message is coming from my impression is, that it is this hivision-node/background polling-process.. do you use websockets...? maybe the log-timeout-erros from node-red are this websocket-timeouts.. how can i get closer/deepter to the cause of the timeout. can you improve message-detail?

can you test/confirm?

image

ozett commented 4 years ago

"connection close"-message https://github.com/nayrnet/node-hikvision-api/blob/master/hikvision.js#L272

if it comes from the old-api, maybe you really fork it an improve for some more status-codes?

ozett commented 4 years ago

time-out issue...

if it comes from the old-api, https://github.com/nayrnet/node-hikvision-api/blob/master/hikvision.js#L50

maybe worth to improve logging of connection close, reason, and maybe configurable time (30s hardcoded, i guess)

a pity, i try to read code, but i am not able to write myself...

ozett commented 4 years ago

six hours later, 2 up,2 down -- out of 4

whats going on? hints how to debug?

image

ozett commented 4 years ago

i had these errors also in the node-red log:

30 Oct 15:24:13 - [error] [hikvision-alarm-in:catW115] Connection closed!
30 Oct 15:24:13 - [error] [hikvision-alarm-in:strare112] Connection closed!
30 Oct 15:24:13 - [error] [hikvision-alarm-in:catH116] Connection closed!
30 Oct 15:24:13 - [error] [hikvision-alarm-in:strali117] Connection closed!

not very stable in the long-term ... ๐Ÿ˜‰ the node-red restart fixed it... (must be something with polling, netsocket/websocket or the like?) ... no clue...

henry-spanka commented 4 years ago

We do not use websockets but keep alive messages to keep the connection open. Unfortunately the HTTP library only sends a 'connection closed' notification. I don't know why the connection is closed. If both the cam and your server are wired I don't know what the problem could be. Either your firmware works differently and after a few hours drops the connection (not sure if we can do anything about it), or your connection is unstable, but you said you use wires.

What I could do is to automatically try to re-establish the connection when it is closed. This will not fix this issue but it will probably reconnect within a sec (or more depending on how long it takes for nodered to detect that the connection dropped). As we send keep alive messages every ~60~ seconds it may take up to ~60~ seconds for the connection to drop. Not sure if you're fine with this but I could at least implement this if you like. Meaning you may lose up to ~60~ seconds of notifications/alarms.

Update: How long does it take for the connection to drop after nodered establishes the connection? Can you please provide exact timestamps. If possible check multiple times to see if there is a pattern (e.g. always after 2 hours).

ozett commented 4 years ago

i have to figure something out to get dop-events/connection-loss with timestamps and report. last time it seemed slowly and hidden to stop..but i will find a logging-way.

about 60s. for re-connect: i guess that will be fine, ๐Ÿ‘

as i have only 6-8 events during 12 hours, means that there is mostly on a day no event to report,

so in 60s or more for a re-connect i would probably only loose 1 event - and only if that event would be incidentally at the same time with the dropout. which is highly unlikly,

means: it would be fantastic to have stable longterm connection with a re-connect (and a possibility for log-event of the re-connect)

๐Ÿ‘ ๐Ÿ‘

henry-spanka commented 4 years ago

Actually I don't know how long it takes that's why I edited my comment.

I implemented automatic reconnect with exponential backoff in commit 4b38301bbeba9500796f0569f6a2dffb91c8199a.

Please update and restart again :)

Please post your log running node red for a few hours. It should log when it the connection drops and automatically reconnect so I can see how long the connection is established before it drops and if it's consistent.

ozett commented 4 years ago

ok. think i would be a helpful feature, if one could trigger some hearbeat-infomation for the connection to debug. otherwise in the node-red-gui there is nothing to see, if the connection dies in background.. (i will test new version and report my findings)

image

henry-spanka commented 4 years ago

There is no need for any debug nodes. If the connection drops the node icon will show "disconnected" and the icon will be red. Once it reconnects it will be back to "connected" (green). You can see the individual disconnect/reconnect messages in the node-red log.

henry-spanka commented 4 years ago

... and you should be able to catch these errors with a catch node and attach it to a debug node if you want to. (although you only get the disconnected, not the reconnected message).

ozett commented 4 years ago

catch-node...never heard...i will investigate.... i head the imagination one could get some details of the connection to see it dying...

you seen the big-timer-node from pete scargill? it sends alive-messages...configurable...good to note in the debug, that there is something running..

but i will investigate catch-node

ozett commented 4 years ago

I implemented automatic reconnect with exponential backoff in commit 4b38301.

Please update and restart again :)

how can i check if i have the new version? Update: I dont fully understand the code, but what about exposing the error-count to debug (if it ever happens..) ๐Ÿ˜„

image

ozett commented 4 years ago

any idea where i see the log from the underlying hikvision-api? also in the node-red log? or in the syslog? or...?

(i found in the api this log-code..maybe it helps to observe this.. as i had 1 lost event (received email, but node-red did not notice)

https://github.com/henry-spanka/node-hikvision-api/blob/master/Hikvision.js#L247

if (this.logToConsole) {
                        console.log('    Skipped Event: ' + code + ' ' + action + ' ' + index + ' ' + count);
ozett commented 4 years ago

just to debug i came up with the idea to grab the output of uri = '/ISAPI/Event/notification/alertStream' from the cam with the http-request-node.

did not work out that simple, but i still have hope, reading here they did it too: https://forum.logicmachine.net/showthread.php?tid=1630

the api-code seems to do the same in an loop. maybe i can build this in node-red itself with the http-request-node.

but i am a bit clueless, why i dont get a response. what am i doing wrong? any idea to test? maybe curl?

ozett commented 4 years ago

one step further...it worked with error in IExplorer

image

Update: Now what? ๐Ÿ˜ธ

henry-spanka commented 4 years ago

Catch node should log all errors. Additionally you will find the errors in the node-red log (as you did before).

The version number will always be the same as I have not released a new version yet. However the code does change if you reinstall again from the master branch.

You need to use basic authentication to access the URL but the request will never complete because you basically subscribe for events. This is already implemented in this plugin so I don't see any point of manually doing this with node red. That's exactly what this plugin is for. I can only say that for my cam it works so either you have some network issues or your cam behaves different.

My model for reference: DS-2CD2155FWD-I, V5.5.61 build 180718

ozett commented 4 years ago

my goal was to test another way to get event information, besides the plugin. another way would be via onvif.

if i had sucseeded, i had tried to configure the polling more lazy. maybe this is an idea to make the polling-value configurable in a config-node? (now on to check the runtime/stability of the plugin over night) ๐Ÿ˜„

henry-spanka commented 4 years ago

There is no polling. Data is received in real-time.

ozett commented 4 years ago

ok. i looked up polling/push in https, there are some RFCs... thats new for me, so i let the node-red run the whole day...without any problems.

just for a moment i disabled some telegram-bot nodes and now i see one camera disconnect shouldnt the camere re-connect by itself after 60 seconds?

image ---Update: camera is reachable, webgui is full responsive, but the node-red connection is not re-established.. i will try to restart node-red completly as it help at last..

ozett commented 4 years ago

image

it is not-reconnecting by itself... but that would be fantastic. what to do?

ozett commented 4 years ago

OOOps image

ozett commented 4 years ago

i restarted all a second time. that worked now.. Something to improve?

ozett commented 4 years ago

i need the hikvision-in node to send images via "node-red-contrib-telegrambot" as i configure the telegram-bot, i have the impression that this affects the connection to the hikvision-cams.

actually after runtime overnight i reconfigured the pollingtime for the telegram-bot to 600, the hikvision-api throgh this error-message:

image

henry-spanka commented 4 years ago

Fixed in commit a36dffde2f1aa83d2024739ac65528314147776f. Please update the plugin and try again.

ozett commented 4 years ago

my impression up to now was, that i run longer stable. last improvement for re-connect may helped. i will update, observe, log and report, Sir!

henry-spanka commented 4 years ago

The reconnect does not work as It crashes when trying to reconnect. Logically there can't be an improvement unless you did something else :)

ozett commented 4 years ago

i disabled lots of unused elements, i uninstall some testing-nodes, i changed some flows for telegram-bot (polling may interrupted), i upgraded my unifi-wifi controller and APs... .... ๐Ÿ˜„ but i will check out your code now ----Update: npm install henry-spanka/node-red-contrib-hikvision#master node-red-contrib-hikvision@1.1.0 (may a litte version jump?)

henry-spanka commented 4 years ago

Do you still have disconnects or can we close this issue now?

ozett commented 4 years ago

i cannot say, that i observe dissconnects from your hikvision-api-node. i had some node-red restarts while fiddling with the hikvision subscrition and copying hikvision-nodes, but i did not investigate further and for now i would say my node-red is running stable enough.

in two words: close it.

and thanks a lot for helping with this

henry-spanka commented 4 years ago

Thanks ๐Ÿ‘

ozett commented 4 years ago

i have spourious disconnects from one of my hikvisons (DS-2DE2A404IW-DE3) still. but today i saw, that there are 4 minor firmware upgrades. i will upgrade to the latest 5.6.14 and see it the situation impoves. i will report.

ozett commented 4 years ago

the release-notes state the version of the isapi-interface. maybe someting to get into the hikvision-node? reading and showing of the isapi-version?

2.2 Protocol Versions
๏ผˆ1๏ผ‰Onvif version: 17.12
๏ผˆ2๏ผ‰ISAPI version: 2.4

http://www.hikvisioneurope.com/portal/portal/Technical%20Materials/01%20%20PTZ%20Camera/00%20%20Product%20Firmware/R7%20series%28DS-2DExxxxW%2CDS-2DFxxxxW%29/Baseline%20Firmware_IPD_R7/V5.6.14_Build190826/PTZ%20Camera%20V5.6.14%20Release%20Note%20--R7%28external%29.pdf