Media Source support

[xannor]

I am working on an enhancement to add media source support for viewing VODs from the cameras. I have forks of this repo and the reolink repo and made some preliminary changes to support it.

I have it working in a dev container up to the point where I can see the entries for the current month, but I cannot test well because I cannot get FFMPEG to work in the container. If it was just changes to the integration I could just swap your repo with mine and test on my actual install, but I don't know how to force it to download the modified reolink dependency.

Any suggestions on how I can test to make sure the video playback works?

[fwestenberg]

You can just overwrite the reolink_dev custom component. Also, the reolink python library which is used can be overruled, but this is a little harder to do. One of the ways is to place the folder in your custom components also, and change the references to this one.

[xannor]

Ok i have a preliminary version working. I feel it still needs some cleanup before it can be considered usable. 1) for the Media Browser component I am giving it a name, but I do not know how to use the languages to provide the proper name, right now I am manually providing the english name. 2) My 510 camera has an issue with playbacks, but it has this issue with VLC also so I have a support ticket in with REOlink, I have tested with a 520 and 511 and both work, though more testing is needed. 3) Right now I only pull the current month as you have to search by date range and I am not sure how far back to attempt search, this might need to be a config option. 4) Because of the search I have horrible folder names and pathing, right now it is Camera Name/Year/Month/day/start time duration.mp4 5) I need to provide some sort of static image for the folders, and thumbnails for the videos. 6) I asked REO and there is no way to thumbnail the videos other than pulling the stream and pulling a frame. I believe this is doable but I do not think it would be very fast and I dont know if I can have tell the client side to refresh 7) Various optimizations and clean up are needed, I do cache some entries but I dont have a good validation/voiding method.

I am not sure how you would like to proceed or if you want me to wait until I have some of these issues worked out.

[fwestenberg]

Hi, thanks for working on this. I only now understand what you're trying to accomplish. Very interesting! I don't know what you need exacly, but to me it sounds like this should become a separate component perhaps, or maybe even a addon? But again, I didn't see the coding and how this works.

[xannor]

Though, in theory it could be a separate component, this just an implementation of the Media Source platform, to expose the recorded videos on the camera.

In my first post I provided links to the two repositories and branches that I made the changes to, Mostly I had to add a search function to the reolink library, and the platform implementation to the integration.

Mostly I was just looking for thoughts and feedback. If this is not something you are interested in I can leave it in my fork.

[fwestenberg]

Don't get me wrong, I really like the idea of the media browser for Reolink. I just do not have an integration to compare with. And parallell to all the new development, I am trying to get this Reolink integration accepted as a HA components. And the more functionality I add, the more complex it will become. But for this custom component (reolink_dev), please send me a pull request as soon as you thing it is stable. Can you show (some pic's) of the result of your code? I'm very curious!

[xannor]

It seems the media browser is relatively new, the only two official components that support it are the netatmo integration and the xbox integration.

Code-wise, what I changed can be seen here for the library and here for the integration.

I will try to work on it some more next weekend, and get you some screen shots so you can see what the client side looks like.

[xannor]

I got it to a point I am happy with so I created the pull requests, #173 for here and #27 in the reolink repo.

The only other thing I would like to be able to do is some sort of way to match a recording to a motion event, but I am not sure how to best approach that.

Also there is some potential for exposure, though I did my best to minimize it. In order to generate playback thumbnails on the fly I needed to create a view that can be called to generate them, however at the point in the platform where I can build its url, I do not have enough context info to make an authenticated url, so I had to disable auth on the view and instead I use a token to verify that my code generated the request. This is similar to how webhooks work, but I figured I should let you know about it beforehand.

[fwestenberg]

It's merged and released!

[fwestenberg]

How should I use this feature? I checked the media browser tab, but no camera devices there.

[t0bse]

Logger: homeassistant
Source: custom_components/reolink_dev/typings.py:28 
First occurred: 20:39:46 (1 occurrences) 
Last logged: 20:39:46

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/integration_platform.py", line 49, in async_component_loaded
    await _process(event.data[ATTR_COMPONENT])
  File "/usr/src/homeassistant/homeassistant/helpers/integration_platform.py", line 30, in _process
    platform = integration.get_platform(platform_name)
  File "/usr/src/homeassistant/homeassistant/loader.py", line 499, in get_platform
    cache[full_name] = self._import_platform(platform_name)
  File "/usr/src/homeassistant/homeassistant/loader.py", line 504, in _import_platform
    return importlib.import_module(f"{self.pkg_path}.{platform_name}")
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/config/custom_components/reolink_dev/media_source.py", line 39, in <module>
    from . import typings
  File "/config/custom_components/reolink_dev/typings.py", line 28, in <module>
    "playback_events": dict[str, VodEvent],
TypeError: 'type' object is not subscriptable

[xannor]

I just updated my install and I see that error too. I did not see this in development, and I think that the next release of HA is bumping to 3.9, it could be that the current release is on 3.8, and you have to use the Dict for typings, in 3.9 you can use the lowercase one. I will fire up my dev container and see which version it is using.

[xannor]

Ok I got a proper dev enviroment setup and I fixed and tested the error. #176 is the pull I created for it.

[fwestenberg]

It's working now! Thanks!

[xannor]

I inadvertently broke something else, I accidentally set the minimum motion off delay to 1, which on my setup I have set to 0, and is causing the options to not save. Not a major issue, unless someone else has it set to 0, and a simple fix that I will submit shortly.

[xannor]

This is a minor correction so unless you have other changes you could just release a 0.16.1 or wait for a more changes for a 0.17.

177

[fwestenberg]

Not very important indeed. I'll create a new release as soon as there is something more interesting to include. Do you have any idea why thumbnails are generated very slow? At this point, it even stopped loading the list at all.

[xannor]

That is an issue, and why I defaulted it to off. Right now to generate thumbnails, I use the FFMPEG and have it read the rtmp stream from the camera and to a screen cap at some point (I defaulted to 6 secs). This was moderately acceptable on my dev machine, but I figured it would be slow in production (I use a pi4) so I defaulted to off. It also does not queue, so it will spawn as many worker threads as thumbnail requests. I probably need to take a page from the generic camera and create a worker thread, either one per camera, or a fixed number, to handle the requests and process the thumbs. I also need to fine a good way to short term cache them, right now they are cached in memory and could eat up a lot of memory if a lot of thumbs are generated.

[xannor]

I just did a quick insertion of task management into my dev environment, only allowing 1 thread per camera for thumbnail processing. It wasn't fast but I generated thumbs for a day that has about 130 events on it in roughly 3 minutes. I think the reason it was failing is overloading the camera and HA system with requests. I just need to work out how to best handle holding these, either purely in memory or via file cache, and I will have another push ready.

[t0bse]

thanks for updating! where can i find my recorded videos? i think you should describe that in a short text

[xannor]

This change added Media Source support to the integration. This allowed the Media Browser component of Home Assistant to "see" the videos available on the cameras. You can get to this via Media Browser -> Reolink IP Camera in the left hand navigation. You will then see 1 "folder" per installed camera. Under a camera you will again see 1 "folder" per day that has recorded events, labeled via the date of the event. The "default" range is two months, as the camera has to be "searched" with a date range to get the recordings. You can change this range in the options for the camera. In the day folder there will be one video per event, labeled as the start time of the event and the duration of the recording.

Right now everything is sorted in the order I receive it from the camera, which I believe is ascending order, but this could possible be different per camera. I used the YYYY-MM-DD and 24hr time formats for now to avoid language issues and to keep the visual sort consistent. Ideally I would like to use either the camera settings, or the HA instance settings to provide a more "friendly" display format but I have to figure that part out still.

[t0bse]

aaah nice, works for me :) awesome feature, great with thumbnails! nice work :)

maybe it could be an option to set it to "all files in one folder" sorted by filename? could be a good solution in combination with just all files of 1 month ago my 8gb sd card is capturing the last 4-5 days only, so "all in one" would be the best option.

[xannor]

I'll think about that, right now I use a quicker method of the search that just gives me the days that have events per month, thus why the the months option and the view videos by day approach. I only fully search one day at a time, which with cameras that have large storage and a lot of events (or an NVR) keeps the load down.

[t0bse]

works fine on ios and android app :-) thumbnails aren't THAT slow on my NAS, but is it possible to save them? every device is rendering the thumbs for itself now, would be much quicker.

[xannor]

I have been mulling over how to persist them. Right now they are held in memory and will persist until you reload HA.

[xannor]

I have a basic fix for the thumbnails, both with generation and caching. I don't have it to a point I am happy with, but if anybody wants to test it, replace the media_source.py with this file and reload your ha instance. I would like some feedback as to how well this works, or breaks, an install.

Right now it will only generate a thumbnail one at a time per camera, and it will save them under config/.storage/reolink_dev/thumbnails/unique_id/timestamp.jpg

I dont have anything at this time to do cleanup.

Update: I am testing this on my rPI and it seems FFMPEG has serious issues on it, so I will have to investigate more.

[t0bse]

you are fast 😃 i can try that tomorrow on my nas system

[andriej]

I can see files, but they are not playable - /api/hls puts 404 to browser (in browsers console), file not found

[Lybbe77]

I've enabled this in options for the integration and can see camera folders. Triggered motion just to test but no files can be seen in the media source folders.

Am i missing something? Just to be sure: No memory card is required in the cameras for this to work, right?

My defined automations are saving streams and snapshots to my other folders as they should so there's no issue with motion detection.

Any clues?

[fwestenberg]

You indeed need a memory card in your camera (or NVR?) for this to work.

[Lybbe77]

You indeed need a memory card in your camera (or NVR?) for this to work.

Thanks for clarifying!

[xannor]

I can see files, but they are not playable - /api/hls puts 404 to browser (in browsers console), file not found

The stream api views respond 404 to invalid requests, you would need a full and correct url to get a response. By not playable, do you mean you get an error or that it does nothing? Which type of install is yours, i.e. hosting environment?

[t0bse]

I have a basic fix for the thumbnails, both with generation and caching. I don't have it to a point I am happy with, but if anybody wants to test it, replace the media_source.py with this file and reload your ha instance. I would like some feedback as to how well this works, or breaks, an install.

Right now it will only generate a thumbnail one at a time per camera, and it will save them under config/.storage/reolink_dev/thumbnails/unique_id/timestamp.jpg

I dont have anything at this time to do cleanup.

Update: I am testing this on my rPI and it seems FFMPEG has serious issues on it, so I will have to investigate more.

working fine for now. how often is it rescaning for new files? today wasn't a folder yet, so i restarted HA and i got it.

also it would be good if you do a "if sequence is shorter then the Pre-record offset do a thumbnail after 1 sec"

[xannor]

I actually noticed that bug yesterday evening. Technically it doesn't rescan, whenever I do a search part of the results is which days of the month(s) in the search have videos, and I use this to generate the folder info. However I only do the prescan on load so it didnt update on day change. I was going to be sneaky and intercept motion events and then just add the day if it is missing, but I was concerned it would add complexity to the code (event binding etc) and as far as I can tell, Media Sources never get torn down once created (even if the integration is unloaded) so I did not want it to hold references to any live components. My solution to this is to just assume the current day could have videos and always include it. This is not something I pushed into my repo yet. This also pointed out another bug in the reolink lib, when a search comes back empty, which I will need to submit a pull for.

I am also working out a fix for the lack of recording capability, and an issue I found with FFMPEG on Raspberry Pi's.

[xannor]

ok, pushed up a new copy of the media_source.py and this version actually runs well on my rPI4! I am manually calling FFMPEG instead of using the ffmpeg component, which uses the haffmpeg library. I dont know if it is its piping or if it is that it is not using async fully, but my manual call directly creates the file and this seems much faster. Would still like you guys to test it before I do a pull request.

[fwestenberg]

Thanks! I'm running the new version, my pc is a HP Elitedesk with i5 and 8GB RAM. Should be enough to generate thumbnails I guess. ;-) But for some reason, after openen an folder with around 30 movies, it just freezes and all other folders even appear empty. Any clue?

[fwestenberg]

Ah wait, there it is, I think:

Logger: homeassistant
Source: custom_components/reolink_dev/media_source.py:437
First occurred: 2:13:14 PM (5 occurrences)
Last logged: 2:13:14 PM

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/reolink_dev/media_source.py", line 428, in _async_save_image
    await self.hass.async_add_executor_job(self._save_image, path, image)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/reolink_dev/media_source.py", line 437, in _save_image
    f.write(data)
TypeError: a bytes-like object is required, not 'NoneType'

[fwestenberg]

And this one:

Logger: haffmpeg.tools
Source: /usr/local/lib/python3.8/site-packages/haffmpeg/tools.py:49
First occurred: 2:13:14 PM (5 occurrences)
Last logged: 2:13:14 PM

Timeout reading image.

[xannor]

That should be plenty for thumb generation. Yeah, this is because the ffmpeg timed out, if you overwrite the media_source.py with the one here that I uploaded about 30 minutes ago, does it work? That is the same issue I saw on my PI which is why I started building a new version.

[xannor]

I think it has to do with how the haffmpeg lib executes ffmpeg, it uses an older method of popen, but pipes both stdin and stderr and I think ffmpeg is overflowing stderr or possibly the stdin is overflowing causing a deadlock and timeouts. in my new code I manually call ffmpeg via asyncio and have it output to the file location.

Or is that I have ffmpeg delay for a few seconds to read into the vidstream, whereas the lib expects just the first frame grabbed, and it just takes it to long with the piping.

[fwestenberg]

That one works a lot better. Not lightning fast, but a big improvement! 👍

[xannor]

I just realized I didn't push my latest version, so please re-download the file and check again.

[xannor]

The fastest it would run would be at whatever offset you have the thumbs at (default of 6) since it has to read the stream that far to grab the right frame.

[xannor]

I dropped my other pull for thumbnails and started a new draft one. I was not happy with the direction I was heading with it, and after some thought I think I have a better approach. I have it in draft #180 so I can continue with my ideas and incorporate any feedback. Please take a look and let me know your thoughts.

[fwestenberg]

Wow, great job again! How does this work?

[xannor]

I have a preliminary description in the draft pull. I wanted to shift the conversion to there instead of bouncing between this issue and a pull request.

[ScorpionSPB]

Hi, can you please tell me if it is possible to change the location of the recording folder? Currently, Reolink is recording video via FTP to my HDD, which is mounted as / home. Is it possible to change / Media Sources / Reolink IP Camera to / home / FTP? (System: Laptop / Debian / Hassio (docker))

[xannor]

Media Source / Reolink IP Camera is a virtual folder, the folders for the cameras and dates are just in memory names, and the actual videos are streamed from the camera. In the last version, only the thumbnails were stored on the home assistant install. In the next version, you can specify where the thumbnails are held, but the video is never stored or copied to Home Assistant.

[ScorpionSPB]

Thanks for clarifying