elad-bar / ha-blueiris

Integration with Blue Iris Video Security Software
179 stars 42 forks source link

Cameras Stop Working #217

Open bwthor opened 9 months ago

bwthor commented 9 months ago

I've had the BI/HA Integration running for quite some time, and have had zero issues. This past week I've started losing all my cameras after a number of hours. BI is working with no issues. In order to get them working again, I need to Reload the Integration, and then it works just fine again. Logs below show some issues.

I'd be more than happy to look at other areas for logs and such if someone can point me in the right direction. HA is updated to the latest and BI is as well (5.8.1.1).

2023-11-26 17:10:54.949 ERROR (MainThread) [custom_components.blueiris.camera] Error getting new camera image from ThorNVR Front Door: [Errno 104] Connection reset by peer
2023-11-26 17:10:54.955 ERROR (MainThread) [custom_components.blueiris.camera] Error getting new camera image from ThorNVR Backyard: [Errno 104] Connection reset by peer
2023-11-26 17:11:05.046 ERROR (MainThread) [custom_components.blueiris.camera] Timeout getting camera image from ThorNVR Front Door
2023-11-26 17:11:14.477 ERROR (MainThread) [custom_components.blueiris.camera] Timeout getting camera image from ThorNVR Garage
2023-11-26 17:11:14.481 ERROR (MainThread) [custom_components.blueiris.camera] Timeout getting camera image from ThorNVR Backyard
2023-11-26 17:11:15.143 ERROR (MainThread) [custom_components.blueiris.camera] Timeout getting camera image from ThorNVR Front Door
2023-11-26 17:11:21.337 WARNING (MainThread) [custom_components.blueiris.api.blue_iris_api] Request #0 to BlueIris (http://192.168.0.12:85) failed, Data: {'cmd': 'camlist', 'session': '6ae0456931407055795c6ca534033ec5'}, Response: {'result': 'fail', 'session': '6ae0456931407055795c6ca534033ec5', 'data': {'reason': 'Invalid session'}}
2023-11-26 17:11:21.450 WARNING (MainThread) [custom_components.blueiris.api.blue_iris_api] Request #1 to BlueIris (http://192.168.0.12:85) failed, Data: {'cmd': 'camlist', 'session': '6ae0456931407055795c6ca534033ec5'}, Response: {'result': 'fail', 'session': '6ae0456931407055795c6ca534033ec5', 'data': {'reason': 'Invalid session'}}
2023-11-26 17:11:21.527 ERROR (MainThread) [custom_components.blueiris.managers.home_assistant] Failed to async_update, Error: 'str' object has no attribute 'get', Line: 196
kramttocs commented 9 months ago

Did you take a BI update or HA update around the time it started? Let's focus on trying to find what changed. The integration hasn't changed for quite some time so it's unlikely to be the root cause though possible it needs changed to meet the change of something else.

bwthor commented 9 months ago

I have updated both within the last week as I typically do when updates are released but I can't pinpoint when it started for sure. I know I should leave well enough alone, but I never learn my lesson.

But like anything else, as soon as I posted this, I haven't had any issues. I'll post back if I do, but it did fail me 4-5 times over the past few days.

kramttocs commented 9 months ago

No worries - I do the same. Trial by fire :) Yeah, let me know and we can see what we can find.

bwthor commented 9 months ago

They stopped working about an hour after my post last night. A few hours ago I deleted the integration, and re-added it using the local BlueIris IP address and port. I may have used the URL before, but not sure. Anyway, got it up and running again, and I see it is down again. Any ideas what logs I should be looking at?

Editing to add, they never just start working again. Only way to make them work is to reload the integration. Restarting BI does nothing.

kramttocs commented 9 months ago

Same sequence of errors as before?

kramttocs commented 9 months ago

And I know you said it was working fine in the past so a BI config isn't likely but just to check as a comparison- you have the authentication settings configured as specified here? https://github.com/elad-bar/ha-blueiris/blob/master/docs/blueiris-server.md

bwthor commented 9 months ago

Yes, same sequence as before. No issues with BI, but on my panel, the four cameras I have just go blank. Connecting from another computer, there is no camera feed either, so not isolated to the computer running the touch panel.

I'm pretty sure I had things set up like in the link you shared. Definitely was using a separate user on LAN only. Not sure about the Require login from Non-Lan Only setting, because I think my proxy was messing with that setting. This time around I was using my regular admin login, but will switch that back and see.

Hard thing is that I can't replicate this, I just have to wait until it fails. I'd assume something should be in the logs, but I'm not seeing anything that makes sense.

bwthor commented 9 months ago

So, again after my last post I have had zero issues. Seems completely random.

kazzaw commented 9 months ago

Also having this issue - Cameras seem to go offline through the integration. Reloading the integration appears to bring them back up. I notice cameras fo offline more frequently when changing server/camera settings. Usually it would recover, however it would appear from a recent blueiris update this may have changed.

kramttocs commented 9 months ago

I am seeing this also - I don't use the video streams in HA so I added a couple when @bwthor reported it. What's interesting is that the mqtt sensors still work correctly - I still get motion detected in HA even though the camera entity isn't showing video.

bwthor commented 9 months ago

I am really hesitant to post this, but I think my cameras have been up for four days. I have changed nothing since the last time they stopped. Now they will quit in a few hours...

kazzaw commented 9 months ago

Yeah I have also seen the same… if it stops working I’ll blame you

stevemcan commented 9 months ago

I have the same although one camera continues to work, all the other just go blank. reloading the intergration brings them all back.

wmichael3 commented 9 months ago

Hey, I've been running this integration for months without a burp. I know, not helpful, but in the early days I recall a few burps and what I ended up doing was to exempt the HA IP address from authentication to BI. You do that with the carat (^) before the IP under the Web Server advanced settings | Limit access by IP address. Just a thought. I do understand why some would not like that approach, but I thought it safe enough for my world and then I have the secure session keys and login page enabled for all other IPs.

DaveStLou commented 8 months ago

I'm seeing the same issue. Similar to others, mine has been running fine since originally installed but now all cameras go blank. I have found that disabling and re-enabling the integration gets it working but that's not a good workaround.

dkirby-ms commented 8 months ago

This started happening to me yesterday, right after an updated to HAOS 12.2. BI integration shows most of the entities of the cameras are "no longer being provided by the Blue Iris integration" but the profiles and schedule entities still work.

Edit: This was my own issue. When I upgraded the BI integration had to be reconfigured and each of my cameras and its entities re-checked in the BI config page.

kramttocs commented 8 months ago

Is everyone getting an invalid session error when this happens? Best guess is something on BIs end changed since the integration hasn't and I wouldn't expect an HA change to impact authentication with BI. Opening a request to BI to see if anything has changed recently around that would be the best initial option.

permster commented 8 months ago

You can actually reproduce this issue fairly easily. Stop the Blue Iris Service, wait until you get a "failed to connect" message in the HA system logs (may take a few minutes), then start the Blue Iris Service. Wait a few more minutes and you should get the "invalid session" system log.

I actually built an automation to reload the configuration of a single camera which seems to fix all cameras as a result. This requires the fire_event: true in your configuration.yaml.

alias: General - Blue Iris Reload
description: ""
trigger:
  - platform: event
    event_type: system_log_event
    event_data:
      name: custom_components.blueiris.api.blue_iris_api
condition:
  - condition: template
    value_template: "{{ '\\'reason\\': \\'Invalid session\\'' in trigger.event.data.message[0] }}"
action:
  - service: homeassistant.reload_config_entry
    data: {}
    target:
      entity_id: camera.blue_iris_courtyard
  - service: notify.mobile_app_android_phone
    data:
      message: Blue Iris integration restoration has been attempted.  Please verify!
      title: Reload Blue Iris integration
mode: single
DaveStLou commented 8 months ago

As a stopgap I set up the automation from @permster but for some reason it didn't trigger even though I had two events (below). Any idea why?

Request #0 to BlueIris (http://10.42.8.50:81) failed, Data: {'cmd': 'camlist', 'session': '37d8168405e63dcc1886083a466f7039'}, Response: {'result': 'fail', 'session': '37d8168405e63dcc1886083a466f7039', 'data': {'reason': 'Invalid session'}}
Request #1 to BlueIris (http://10.42.8.50:81) failed, Data: {'cmd': 'camlist', 'session': '37d8168405e63dcc1886083a466f7039'}, Response: {'result': 'fail', 'session': '37d8168405e63dcc1886083a466f7039', 'data': {'reason': 'Invalid session'}} 

I ran it manually and the action reset all the cams but I'd like to get it to trigger without intervention until this issue is resolved.

permster commented 8 months ago

Did you enable the system log fire_event: true in your configuration.yaml as documented here. Make sure your restart HA after the config update.

DaveStLou commented 8 months ago

That was it. Thank you!

Kedryn commented 8 months ago

I have same problem (way longer than 3 weeks ago). cams stop working and come back absolutely random. What i've noticed is that cams disappears from the integration, when i get a timeout on the Ha lovelace page that shows them. So it's something between that and HA. Also because the the Tapo app shows them online.

Is it normal for cams to disappear and come back again from the BI integration? image i saw "devices" go down as low as 2 for few seconds.

tcouey commented 8 months ago

Any updates on this? I've been having the same issue for a couple of months now. I was hoping the latest update to the integration and HA 2024.1.0 would fix it, but it has not. They still all go offline after some amount of time (a few hours). No issue that I can find in Blue Iris.

kramttocs commented 8 months ago

No real updates. BI support told me there was a change:

Yes if you send an invalid session to the /video? endpoint, it will [now] return invalid session. Previously, it would redirect to the login page. That didn't make sense and our UI3 developer asked for this change.

But I haven't found a connection between it and this behavior.

kozoke commented 7 months ago

Same issue. Started when I updated blue iris to the most current release. I rolled back to the 12/5/23 version (very easy) and it appears to be working fine. Thanks for the ack above. Look forward to an update. :-)

SpudsterZ commented 7 months ago

Ditto here, HOWEVER everything has been stable for the past couple days after the last restart of the integration. I do see BI upgraded to 5.8.4.4 on the 16th, so here's to hoping the that resolved this issue.

PrestonMcAfee commented 7 months ago

Had this issue today, HA 2024.1.6, BI 5.8.5.7. Reloading the BI integration fixed for me. Updating BI to 5.8.6 to see if it helps. The update, which briefly shuts off BI, did not affect camera access.

I haven't yet gotten to using the BI integration to do anything other than a quick way of looking at cameras, as I find the BI web interface works well, but I have plans to turn on lights when motion is detected in BI, and this loss of connection would put a crimp in that.

Thanks, permster, for a workaround!

kramttocs commented 7 months ago

@PrestonMcAfee The camera stream is separate from the mqtt traffic. You will use mqtt for the motion detection so you'll be good there.

If you have any questions on the mqtt part, check out the guide here in GitHub (on mobile so can't easily link it).

PrestonMcAfee commented 7 months ago

Thanks! I still have a lot to learn.

On Sat, Feb 3, 2024 at 12:07 PM kramttocs @.***> wrote:

@PrestonMcAfee https://github.com/PrestonMcAfee No, the camera stream is separate from the mqtt traffic. You will use mqtt for the motion detection so you'll be good there.

— Reply to this email directly, view it on GitHub https://github.com/elad-bar/ha-blueiris/issues/217#issuecomment-1925443739, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKANTXLPN6UGWONKKTGHXE3YR2KIZAVCNFSM6AAAAAA74ALH2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVGQ2DGNZTHE . You are receiving this because you were mentioned.Message ID: @.***>

kramttocs commented 7 months ago

You bet! Here is the link for that manual: https://github.com/elad-bar/ha-blueiris/blob/master/docs/blueiris-server.md

SpudsterZ commented 7 months ago

Ditto here, HOWEVER everything has been stable for the past couple days after the last restart of the integration. I do see BI upgraded to 5.8.4.4 on the 16th, so here's to hoping the that resolved this issue.

Well, it hasn't been 100% since I last posted the above, but it's only been about 3 times where the cameras went wonky. In one occasion, the integration reload didn't work, requiring a full HA restart.

So not perfect, but far and above better than the near daily issue that was once seen.

trailiajohn commented 6 months ago

I'm also having this issue. The automation above sometimes works, but even manually reloading the integration I couldn't get my feeds back without a full HA restart.

kramttocs commented 6 months ago

That seems like a different issue (or multiple) as restarting the integration generates a new bi auth session. Did you see anything in the bi logs? Like an invalid login or anything?

trailiajohn commented 6 months ago

BI logs do show the logins when rebooting the integration without any error on the BI side. MQ works great, but cameras take a full HA restart to resolve the problem in my Dashboard.

SpudsterZ commented 6 months ago

Just had a situation where the automation above restarted the integration, but it did not recover, requiring a full HA restart. Checking BI logs shows this happened when BI performed an automated update and restarted itself.

jatgm1 commented 6 months ago

So I also have this issue. It's super annoying. I use them for motion sensors.. none of my lights are working now and it's super annoying, I smashed my toe on a door frame in the dark.. how can I make this work for now? Or like just in general.

kramttocs commented 6 months ago

The camera stream issue should be completely isolated from mqtt which is what is used for motion detection.

jatgm1 commented 6 months ago

Oh crap it did a backup and some stuff stopped working i think a restart fixed it, correlation is not always causation, my apologies, disregard that part. The cameras do blank out though and won't come back without reloading the integration.

kramttocs commented 6 months ago

No worries - hope the toe turns out ok :)

jatgm1 commented 6 months ago

Has anyone figured out what's wrong and how to make it work now?

rlsgit commented 5 months ago

Same here...appears to be a logout occurring...I'm going to dig deeper...Just moved from Shinobi to BlueIris, and this hurts...

wmichael3 commented 5 months ago

If you exempt your HA IP address from requiring authentication in BI all of these problems will disappear.

rlsgit commented 5 months ago

If you exempt your HA IP address from requiring authentication in BI all of these problems will disappear.

Well, that was harder to find than I thought, but will keep an eye on it...no to the other erroneous log entries the integration sends...

wmichael3 commented 5 months ago

Sorry, should have added detail: in Blue Iris main settings: (cog icon at top left). Web Server, Advanced, Limit access by IP address, in the box enter a carat "^" followed by the IP of your HA instance. For example ^192.168.1.10

Since doing this it has been perfect. Not a solution, but a workaround.

bwthor commented 5 months ago

If you exempt your HA IP address from requiring authentication in BI all of these problems will disappear.

I use NGINX Reverse Proxy, and I seem to recall when I set this, it allowed anyone to log on, even from outside. It was 6+ months ago and my memory is junk, but that seems to be the issue. Either way, double check the access from outside if you do set this.

rlsgit commented 5 months ago

Sorry, should have added detail: in Blue Iris main settings: (cog icon at top left). Web Server, Advanced, Limit access by IP address, in the box enter a carat "^" followed by the IP of your HA instance. For example ^192.168.1.10

Since doing this it has been perfect. Not a solution, but a workaround.

Thanks, I did +192.168... and will keep an eye on it.

rlsgit commented 5 months ago

If you exempt your HA IP address from requiring authentication in BI all of these problems will disappear.

I use NGINX Reverse Proxy, and I seem to recall when I set this, it allowed anyone to log on, even from outside. It was 6+ months ago and my memory is junk, but that seems to be the issue. Either way, double check the access from outside if you do set this.

So, I'm home only, not commercial, and do have access into my system by rule and secured for me. But I agree it's a bit scary. I'm a dev by nature and have written some HA stuff, so may poke around a but. I also hat log entries that indicate potential issues. I've done ALOT to clean my system up and hate seeing some integrations that just generate "noise"...

trailiajohn commented 5 months ago

BI documentation says you need an account named "admin", which I added, but it's still not working for me. I always end up on the login page.

johnjoemorgan commented 4 months ago

Sorry, should have added detail: in Blue Iris main settings: (cog icon at top left). Web Server, Advanced, Limit access by IP address, in the box enter a carat "^" followed by the IP of your HA instance. For example ^192.168.1.10

Since doing this it has been perfect. Not a solution, but a workaround.

Tryng this now

johnjoemorgan commented 4 months ago

Sorry, should have added detail: in Blue Iris main settings: (cog icon at top left). Web Server, Advanced, Limit access by IP address, in the box enter a carat "^" followed by the IP of your HA instance. For example ^192.168.1.10

Since doing this it has been perfect. Not a solution, but a workaround.

Screenshot_13

Been working flawlessly for last 3 hours. Both via direct local IP and via ZeroTier. That 2.5 hours longer than any other time in the last 6 months or whenever this problem started

Thanks @wmichael3