search

thirtythreeforty / neolink

An RTSP bridge to Reolink IP cameras
https://www.thirtythreeforty.net/posts/2020/05/hacking-reolink-cameras-for-fun-and-profit/
GNU Affero General Public License v3.0
883 stars 145 forks source link

B800 / D800 primary stream issues? / timezone #140

Closed eisengrau closed 3 years ago

eisengrau commented 3 years ago

Hello,

After founding this piece of software and repo, I am tempted to acquire one of these D800 cameras for home security purposes as a test (maybe more later on if all is working properly). I still consider these the cheapest, and by reading the reviews, one of the best 4K capable cameras for the price (for around €60 on Aliexpress).

However, after I read some of the posts here and on Reddit, and if I understood correctly, there are issues with this B800s (or D800, I assume it is the same HW only different form factor/size) this neolink bridge solution and the primary (4K stream) - in some cases it is not possible to access / monitor or record the pirmary stream (?) only the secondary, which is (HD 1280 x 1024 ?). If this is the case, it is not worh buying this camera, as there are many good Full HD capable PoE cameras (and even 2K from Xiaomi) for half of the price.

My intention is to use these cameras with Shinobi (or Motioneye) and one of my spare Intel NUCs, as a makeshift NVR solution (which I would rather assume that is a more capable solution than Reolink's own NVR, and of course the hardware is ready available), and I would like to avoid purchasing the NVR as well.

What are the current experiences with these issues? It is worth going into this? Are these issues just case-by-case basis, or did I misunderstood something?

Thank You,

QuantumEntangledAndy commented 3 years ago

I beleive the main issue with the HD is bandwidth. Full HD streaming is a lot of MB/s and WiFi tend to get overloaded by it. definitely keep as much as possible done over Ethernet, Maybe cam<-wifi->router <-ethernet->shinobi/neolink machine. Probably best if the shinobi is on the same machine as neolink set it to record the hd to your disk and for live viewing watch the SD.

QuantumEntangledAndy commented 3 years ago

Its good to test your setup and make sure it all works for you.

QuantumEntangledAndy commented 3 years ago

There's also an issue with the docket image and HD that we haven't pinned down so best to run neolink on the metal.

eisengrau commented 3 years ago

Hello,

I think there is a misunderstanding here. I was referring to the primary stream of the B800/D800 cameras, which are - according to the specs and the original product description - are capable to output a video stream in 4K resolution H.265 encoding. That is 4x times more resolution than Full HD.

As I understood, per previous issue requests and over at Reddit, that the HD stream (which is configured as a secondary stream) for the above mentioned devices, are WORKING properly with neolink. The issues are only with the PRIMARY, alas the fully-capability 4K resolution one.

I also understand the bandwith requirements of a 4K stream (albeit with H.265 encoding, which apparently the camera hardware CPU is capable of), and for me - and also the B800 cameras are PoE WIRED ethernet only devices - using copper Ethernet versus Wireless is not a question.

QuantumEntangledAndy commented 3 years ago

Yes I know 4k I just refer to the 2 streams as HD and SD. I believe reolink calls then clear and fluent. There are so many different names for the camera resolutions that it's just easier for this sort of discussion.

Let's start again. I am not aware of any specific issues with these cameras. These cameras are the ones @thirtythreeforty uses and he's the primary developer. The only isuues I can think of without knowing your source of concern are those be related to bandwidth or processing.

Could you perhaps link me to the specific issues that are concerning you.

eisengrau commented 3 years ago

Oh, OK, I undestand now. The confusing part was with the HD (vs 4K) naming.

One example of a mainStream not working, is in this reddit thread: https://www.reddit.com/r/ReverseEngineering/comments/glek0l/hacking_reolink_ip_cameras_for_fun_and_profit/ , but it seems like it is a case-by-case thing.

118 is also quite verbose about this too...

QuantumEntangledAndy commented 3 years ago

The issues in the Reddit are quite old and should have been fixed by code since then.

In issue #118 it is due to the client not keeping up with the packets coming in over UDP probably just too much data. Using tcp would help it not drop packets. The second issue at the end of #118 was resolved in a different thread and it was just a missing dependency.

At this point I recommend you just give it a try and give me a shout if you run into any specific issues.

eisengrau commented 3 years ago

Hello,

I was finally able to test the setup with neolink and a couple of D800 cameras. It is flawless! I'm running neolink under an LXC in Proxmox, and so far is good. The only minor issue is with the camera time setting. It seems that neolink sets the time with -2 hours compared to the actual timezone (set with tzdata).

QuantumEntangledAndy commented 3 years ago

Neolink does nothing special with the time. This is just a docker configuration issue. Please ensure that in addition to the TZDATA you also set the /etc/timezone with appropriate read only mounts. There are numerous guides online for this.

eisengrau commented 3 years ago

/etc/timezone accurately lists the proper timezone of the container. It seems like the service defaults to GMT. It is not a huge deal. Maybe only in the timestamp of the video, However it is possible to set a custom NTP server in the Reolink app of the cameras, and schedule sync as that will hopefully keep the time updated.

QuantumEntangledAndy commented 3 years ago

Managing an NTP is outside of scope of this project. We just use the system time. The issue is almost certainly that the time inside the container is not setup correctly. When you say /etc/timezone is correct. Are you referring to the /etc/timezone of the host machine or the container. I really cannot be sure unless I know more details of your setup but I suspect your quoting the hosts value. If you are sure the timezone is correct You should also check /etc/localtime of the container.

Some details that could be useful to know at this point are for example: are you using your own container or our docker one? Our docker one is alpine Linux.

If you want to set the ntp in our container just setup the appropriate mounts to change the configuration files that set the ntp for alpine linux. But NTP just sets the time not the timezone so this will only be helpful if the /etc/localtime is not right.

Googling around proxmox (never used it) you'll need a mount point configuration line like this

mp[n]: [volume=]<volume> ,mp=<Path> [,acl=<1|0>] [,backup=<1|0>] [,mountoptions=<opt[;opt...]>] [,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>] [,size=<DiskSize>]

If you followed my previous advice you'd have a line like this for /etc/timezone (change [n] to a free number

mp[n]: /etc/timezone,mp=/etc/timezone,ro=1

This mount point should be setup for the containers root user.

You should probably also set one up for the /etc/localtime

mp[n]: /etc/localtime,mp=/etc/localtime,ro=1

Both of these mount when done should ensure that the container matches the host.

You can check the container time by: running a command inside the container such as date.

eisengrau commented 3 years ago

I can confirm that /etc/timezone is in fact the timezone of the LXC container, neolink141 is the host name of it (+ the host has the same timezone). The container is a Debian Buster image.

root@neolink141:~# date Wed May 26 08:37:02 CEST 2021 root@neolink141:~# cat /etc/timezone Europe/Budapest root@neolink141:~# systemctl stop neolink root@neolink141:~# systemctl status neolink

  • neolink.service - Neolink Loaded: loaded (/etc/systemd/system/neolink.service; enabled; vendor preset: enabled) Active: inactive (dead) since Wed 2021-05-26 08:37:21 CEST; 5s ago Process: 1836 ExecStart=/root/neolink --config /root/neolink141.toml (code=killed, signal=TERM) Main PID: 1836 (code=killed, signal=TERM)

May 25 11:48:48 neolink141 neolink[1836]: [2021-05-25T09:48:48Z INFO neolink] cam2: Connected and logged in May 25 11:48:48 neolink141 neolink[1836]: [2021-05-25T09:48:48Z INFO neolink] cam2: Starting video stream subStream May 25 11:48:48 neolink141 neolink[1836]: [2021-05-25T09:48:48Z INFO neolink] cam2: Connected and logged in May 25 11:48:48 neolink141 neolink[1836]: [2021-05-25T09:48:48Z INFO neolink] cam2: Camera time is already set: 2021-05-25 9:48:48 +0 May 25 11:48:48 neolink141 neolink[1836]: [2021-05-25T09:48:48Z INFO neolink] cam2: Camera reports firmware version v2.0.0.17_20042201 May 25 11:48:48 neolink141 neolink[1836]: [2021-05-25T09:48:48Z INFO neolink] cam2: Starting video stream mainStream May 26 08:37:21 neolink141 systemd[1]: Stopping Neolink... May 26 08:37:21 neolink141 systemd[1]: neolink.service: Main process exited, code=killed, status=15/TERM May 26 08:37:21 neolink141 systemd[1]: neolink.service: Succeeded. May 26 08:37:21 neolink141 systemd[1]: Stopped Neolink. root@neolink141:~# systemctl start neolink root@neolink141:~# systemctl status neolink

  • neolink.service - Neolink Loaded: loaded (/etc/systemd/system/neolink.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-05-26 08:37:45 CEST; 5s ago Main PID: 2518 (neolink) Tasks: 5 (limit: 4915) Memory: 7.0M CGroup: /system.slice/neolink.service `-2518 /root/neolink --config /root/neolink141.toml

May 26 08:37:45 neolink141 systemd[1]: Started Neolink. May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] Neolink 5a62bb1df1f6bb01d1d12e9383055c637d60de8b release May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Connecting to camera at 172.100.102.2:9000 May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Connecting to camera at 172.100.102.2:9000 May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Connected and logged in May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Connected and logged in May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Starting video stream subStream May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Camera time is already set: 2021-05-26 6:37:46 +0 May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Camera reports firmware version v2.0.0.17_20042201 May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Starting video stream mainStream root@neolink141:~# systemctl status neolink

  • neolink.service - Neolink Loaded: loaded (/etc/systemd/system/neolink.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-05-26 08:37:45 CEST; 2min 18s ago Main PID: 2518 (neolink) Tasks: 5 (limit: 4915) Memory: 8.6M CGroup: /system.slice/neolink.service `-2518 /root/neolink --config /root/neolink141.toml

May 26 08:37:45 neolink141 systemd[1]: Started Neolink. May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] Neolink 5a62bb1df1f6bb01d1d12e9383055c637d60de8b release May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Connecting to camera at 172.100.102.2:9000 May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Connecting to camera at 172.100.102.2:9000 May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Connected and logged in May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Connected and logged in May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Starting video stream subStream May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Camera time is already set: 2021-05-26 6:37:46 +0 May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Camera reports firmware version v2.0.0.17_20042201 May 26 08:37:45 neolink141 neolink[2518]: [2021-05-26T06:37:45Z INFO neolink] cam2: Starting video stream mainStream root@neolink141:~# systemctl restart neolink root@neolink141:~# systemctl status neolink

  • neolink.service - Neolink Loaded: loaded (/etc/systemd/system/neolink.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-05-26 08:40:14 CEST; 2s ago Main PID: 2529 (neolink) Tasks: 5 (limit: 4915) Memory: 5.2M CGroup: /system.slice/neolink.service `-2529 /root/neolink --config /root/neolink141.toml

May 26 08:40:14 neolink141 systemd[1]: Started Neolink. May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] Neolink 5a62bb1df1f6bb01d1d12e9383055c637d60de8b release May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Connecting to camera at 172.100.102.2:9000 May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Connecting to camera at 172.100.102.2:9000 May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Connected and logged in May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Connected and logged in May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Starting video stream subStream May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Camera time is already set: 2021-05-26 8:40:14 +1 May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Camera reports firmware version v2.0.0.17_20042201 May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Starting video stream mainStream root@neolink141:~# systemctl status neolink

  • neolink.service - Neolink Loaded: loaded (/etc/systemd/system/neolink.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-05-26 08:40:14 CEST; 37s ago Main PID: 2529 (neolink) Tasks: 5 (limit: 4915) Memory: 6.8M CGroup: /system.slice/neolink.service `-2529 /root/neolink --config /root/neolink141.toml

May 26 08:40:14 neolink141 systemd[1]: Started Neolink. May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] Neolink 5a62bb1df1f6bb01d1d12e9383055c637d60de8b release May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Connecting to camera at 172.100.102.2:9000 May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Connecting to camera at 172.100.102.2:9000 May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Connected and logged in May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Connected and logged in May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Starting video stream subStream May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Camera time is already set: 2021-05-26 8:40:14 +1 May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Camera reports firmware version v2.0.0.17_20042201 May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Starting video stream mainStream `root@neolink141:~#

As it is visible in the log output, the logs time stamp is correct, but the INFO timestamp that comes from the neolink service is behind 2 hours (6:00) from the real time in relation to the timezone (8:00). Right before the second restart of the service I have tried changing the time from the reolink windows utility to see if it will persist and neolink will "see" the effective time from the device, It does:

May 26 08:40:14 neolink141 neolink[2529]: [2021-05-26T06:40:14Z INFO neolink] cam2: Camera time is already set: 2021-05-26 8:40:14 +1

I have multiple containers running under various Linux OS images, even though Proxmox has it's own timesyncd. is not necessary to mount the /etc/timezone directive into the container. It is possible to set the timezone from within the containers console itself with any available utility, eg. under Debian "dpkg-reconfigure tzdata" and this will persist.

QuantumEntangledAndy commented 3 years ago

Hmm there's a lot of info there but I think im starting to see what you mean. Thank you for the logs they are most helpful.

Now about the info timestaps. Those are working as intended. We use the depedency env_logger for logs and it formats timestamps using RFC3339 which means UTC. The Z at the end of the time effectivly means in standard time. Not really worth changing that.

I'm not sure about the time setting on the camera itself. Our code tries to get the local time and defaults to UTC if that fails for some reason to identify the timezone. To do this we use this function try_now_local from the time library. I can only assume that it is failing for some reason on your machine. If your up for some testing perhaps you can try and run it on a machine directly and not in a container just to remove that as a variable.

eisengrau commented 3 years ago

Sure no problem. Thank you for your time and dedication as well!

It seems that neolink really has nothing to do with this, it is the clock in the camera software. Especially it will reset after every reboot and it won't keep the time zone neither.

I can try running neolink under windows in a QEMU VM and on a physical dedicated Intel NUC as well. Also, speaking of Docker, I can also try running it in an Alpine Linux LXC container too, which I prefer over Debian. I will get back with more details.

QuantumEntangledAndy commented 3 years ago

How did this go did you every manage to investigate this further?

eisengrau commented 3 years ago

Sorry, I didn't have the time really. But after leaving it running as is, I can conclude that it working as it should, so really no need to investigate further on. Especially the newever version of Proxmox (7) uses chrony as default for timesync.

QuantumEntangledAndy commented 3 years ago

I see, thank you for the update :). Shall we close this then? Or is there something that still needs addressing?

eisengrau commented 3 years ago

We can close it, thank you! :)