search

tfabris / CrowCam

A set of Bash scripts to control and maintain a YouTube live cam from a Synology NAS.
GNU General Public License v3.0
3 stars 3 forks source link

Ensure that CrowCam works on DSM 7.x #77

Open tfabris opened 5 months ago

tfabris commented 5 months ago

Currently the CrowCam code is tested and working on Synology DSM version 6.2.

There is a DSM 7.x version available, and 6.2 will reach end-of-life soon. I received an email that said support for DSM 6.2 will end on October 1st, 2024. I should make sure to upgrade my Synology to version 7 and ensure that CrowCam works on it.

I had an email conversation with Alex Santantonio from https://oldtownhome.com - He's using the CrowCam code to run his OspreyCam on YouTube at https://www.youtube.com/@OldTownHomeBlog/streams - He says that he needed to make some changes to the script to make it compatible with DSM 7.x. I'm trying to get a hold of him again to get some tips from him.

My guess is that his code and my code have diverged enough at this point that a merge would be difficult, but I'd certainly like to see Alex's changes that he made to make it DSM 7.x compatible.

quog commented 1 month ago

One issue: IsServiceUp in CrowCamHelperFunctions.sh uses synoservicectl which doesn't exist anymore in DSM 7.2 There is sudo synosystemctl get-active-status pkgctl-SurveillanceStation.service but it needs sudo which is an issue. I commented out the service check on line 918 of CrowCam.sh

quog commented 1 month ago

Second issue (I'm debugging as I go) - I had to replace authWebCall="auth.cgi?api=SYNO.API.Auth&version=1&method=login&account=$username&passwd=$password&session=SurveillanceStation&format=cookie" with authWebCall="entry.cgi?api=SYNO.API.Auth&version=3&method=login&account=$username&passwd=$password&session=SurveillanceStation&format=cookie" in CrowCamHelperFunctions.sh line 1100

quog commented 1 month ago

One other issue, ping is no longer able to run unless you sudo - you get ping: socket: Operation not permitted Two solutions: chmod +s /bin/ping which I don't like as it will potentially be lost with a DSM upgrade So I changed line 650 of CrowCam.sh to be if nslookup $ThisTestSite 8.8.8.8 >/dev/null as nslookup still works for a user, 8.8.8.8 resolves as dns.google, and nslookup returns appropriately for the if statement. And by having the server parameter, it means internal DNS caching is bypassed.

quog commented 1 month ago

FYI - the stream is at https://www.youtube.com/channel/UCIo6tmAPHtFduP3of7quqOw/live

tfabris commented 1 month ago

@quog - Thank you so very much!

Regarding "IsServiceUp" and "synoservicectl" - I looks like this was originally done as bulletproofing, just to make sure that someone who was running this script truly met the requirements. By which I mean, being on a Synology with Surveillance Station installed and running. I don't think that level of bulletproofing is truly necessary, and since it doesn't work any more, it makes sense to skip it. So indeed I will entirely remove the code which calls synoservicectl.

Regarding the "auth.cgi" versus "entry.cgi" change, and the move from v1 api to v3 api: I'll have to research the difference between those CGIs for the API endpoint. If you have more information about what the difference is between "auth" and "entry", let me know, otherwise I'll have a scrape around the docs about it. Additionally, I'm going to need to figure out why I needed to back down to V1 of the API. I made a note to myself about having had troubles with the V3 API, and I'm going to need to do some experimenting to find out exactly what those troubles were. I should have left myself better reminders, so that I know what to look for.

Regarding the change of "ping" to "nslookup" - This seems like a logical change on the face of it, but I need to understand it in more detail. In particular, I need to be absolutely certain that I'm not getting a cached version of the DNS from my local router when I use nslookup in favor of ping. You say that it should work, and I believe you, but sometimes I've seen other network commands lie to me before and feed me cached data (even when their docs say they shouldn't), and I have to fall back to using actual ping to get real time information. I also don't understand why sudo is going to be required to run ping if the script is running under root. When I set these things up on my synology in my Task Scheduler, the script is already running under root and sudo shouldn't be needed. Are you sure that sudo is really required to run ping, you're not just debugging the script under some lower account? Have a look at https://github.com/tfabris/CrowCam/blob/master/README.md#usage-and-troubleshooting and see if the notes there about running it with sudo and running under root in the task manager are applying to your situation. I don't have V7 installed yet, so I don't know if the task manager works any differently, so you might have more information on this than me.

Thanks again! Let me know if you find anything else.

quog commented 1 month ago

Hi Tony,

Those were the only changes needed to make it work on my client's NAS running DSM 7.2.1. Note: I have the scheduled tasks running as an admin user, but not as root - and that seems to be working (at least so far.) It's only been running for a couple of hours so I will see what happens tomorrow.

And yes, ping requires root or sudo now. The change to using nslookup was primarily because my client often had internet issues, and I didn't want to remove that check altogether. By giving two parameters to nslookup, you are telling it to talk to the DNS server in the second parameter which in my example is outside their network. So unless something like a firewall is proxying DNS, you are pretty much guaranteed to have a network connection if that command works.

I found the new Synology APIs here: https://global.download.synology.com/download/Document/Software/DeveloperGuide/Os/DSM/All/enu/DSM_Login_Web_API_Guide_enu.pdf

Thanks for the great code.

Regards Damian

tfabris commented 1 month ago

@quog - Understood, thanks so much! Your comments are very helpful. Since my NAS is still running DSM 6.x, I don't have a way to test on DSM 7.x at the moment. My intention is to upgrade my NAS to a newer model which is already running DSM 7.x, and then migrate the data and scripts over to it, but I haven't done that yet.

I've updated the code to remove synoservicectl already. Regarding the other two changes...

Clarifying questions:

Thanks again!

tfabris commented 1 month ago

Also, @quog - attached to this message is a test script containing a prototype of the "fixed" version of the WebApiAuth function. Would you mind helping me out, and running this test file from the console and sending me its output? It should run from a Linux or Mac shell prompt, and it should be run from a computer on the same network as the NAS. You have to edit three variables at the top of the file to point to your server running DSM 7.x, and its username, rename the file from .txt to .sh, and chmod it to be executable. The output should look similar this, except yours should try to authenticate to version 7 instead of version 6:

DSM Authentication Test - Querying http://192.168.0.88:5000/webapi to see if it is DSM 6.x.
DSM Authentication Test - Version Response: {"data":{"SYNO.API.Auth":{"maxVersion":6,"minVersion":1,"path":"auth.cgi"}},"success":true}.
DSM Authentication Test - DSM version appears to be 6 - Result: {"data":{"SYNO.API.Auth":{"maxVersion":6,"minVersion":1,"path":"auth.cgi"}},"success":true}.
DSM Authentication Test - Querying http://192.168.0.88:5000/webapi to see if it is DSM 7.x.
DSM Authentication Test - Query did not succeed - Result: {"error":{"code":102},"success":false}.
DSM Authentication Test - Attempting to authenticate to DSM version 6.
DSM Authentication Test - Auth response: {"success":true}.

Script should be attached below. Thanks! DsmAuthTest.txt

tfabris commented 1 month ago

Also, @quog - Can you tell if logging to the Synology log (the function "LogMessage" in CrowCamHelperFunctions.sh) is working on DSM 7.x?

tfabris commented 1 month ago

OK, I believe I've made the necessary changes to the master branch code to make it work under DSM 7.x, as long as it's run as "root".

I was able to do some limited testing on DSM 7.2 by logging into their online "demo" of DSM 7.2 from the Synology main web site. I see that the logging function still works, my tests to detect DSM6 or DSM7 are working, etc. I wasn't able to actually test whether the authentication API command works, though, because when you log into their online demo, you don't know the password for the user "demouser" and it terminates your session if you try to add new users or change the demouser password.

I haven't yet updated the "ping" command, because I have strong feelings about it needing to truly be a "ping" and not some other command. The trick of using nslookup instead of ping is a good trick, but it's not exactly the same thing. In particular, I'd like to have the option to change the configuration so that it pings some other server other than a DNS server to perform the network test. There could theoretically be situations where you would want that. Leaving it as a true ping gives it that flexibility.

@quog - If you are able, please try the new master branch version of the CrowCam code on your 7.2 server and tell me if it's working OK for you. The only important changed files are "CrowCam.sh" and "CrowCamHelperFunctions.sh", and you can leave your configuration files as they are. But you will need to reconfigure the tasks in Task Scheduler so that they run under the user "root" in order for the ping command to work.

quog commented 1 month ago

Hi Tony,

If the scripts had been configured to run from the Synology Task Scheduler as "root" with the "bash" command line (as described in the configuration instructions here), then would the existing "ping" command have worked without modification?

Yes - note not tested, but I could ping as root from an SSH connection to the NAS

After you modified the "WebApiAuth" function to use "entry.cgi" and "version=3" to log in, did the other API commands still work without modification?

Yes - I made no other modifications.

Regards Damian

Damian Douglas-Meyer 0414 419 624 @.***

Damian Douglas-Meyer 0414 419 624 @.***

On Wed, 17 Jul 2024 at 04:33, Tony Fabris @.***> wrote:

@quog - Understood, thanks so much! Your comments are very helpful. Since my NAS is still running DSM 6.x, I don't have a way to test on DSM 7.x at the moment. My intention is to upgrade my NAS to a newer model which is already running DSM 7.x, and then migrate the data and scripts over to it, but I haven't done that yet.

I've updated the code to remove synoservicectl already. Regarding the other two changes...

Clarifying questions:

If the scripts had been configured to run from the Synology Task Scheduler as "root" with the "bash" command line (as described in the configuration instructions here), then would the existing "ping" command have worked without modification?

After you modified the the "WebApiAuth" function to use "entry.cgi" and "version=3" to log in, did the other API commands still work without modification? Examples:

CrowCamHelperFunctions.sh

"IsStreamRunning" "StartStream" "StopStream" "CreateNewStream"

CrowCam.sh

Three instances of calls to "entry.cgi?api=SYNO.SurveillanceStation.YoutubeLive"

Did those API calls continue to work, unmodified, with the "version=1" parameter still in them?

Thanks again!

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

quog commented 1 month ago

Hi Tony,

The tests for DSM version in that script worked, but the auth didn't - which I can't figure out as the same code works in CrowCam.sh - perhaps some other instantiation is needed?

My settings where the same as in the live CrowCam.sh (with the correct password of course.) username="apiaccess" password="REDACTED" webApiRootUrl="http://127.0.0.1:5000/webapi"

DSM Authentication Test - Querying http://127.0.0.1:5000/webapi to see if it is DSM 6.x. DSM Authentication Test - Version Response: {"data":{"SYNO.API.Auth":{"maxVersion":7,"minVersion":1,"path":"entry.cgi"}},"success":true}. DSM Authentication Test - DSM version appears to be 6 - Result: {"data":{"SYNO.API.Auth":{"maxVersion":7,"minVersion":1,"path":"entry.cgi"}},"success":true}. DSM Authentication Test - Querying http://127.0.0.1:5000/webapi to see if it is DSM 7.x. DSM Authentication Test - DSM version appears to be 7 - Result: {"data":{"SYNO.API.Auth":{"maxVersion":7,"minVersion":1,"path":"entry.cgi"}},"success":true}. DSM Authentication Test - Attempting to authenticate to DSM version 7. http://127.0.0.1:5000/webapi/entry.cgi?api=SYNO.API.Auth&version=3&method=login&account=apiaccess&passwd=REDACTED&session=SurveillanceStation&format=cookie DSM Authentication Test - Auth response: {"error":{"code":400},"success":false}. DSM Authentication Test - The call to authenticate with the Synology Web API failed. Exiting program. Response: {"error":{"code":400},"success":false}. Terminated

Weird - I will look at it again later.

Regards Damian

Damian Douglas-Meyer 0414 419 624 @.***

On Wed, 17 Jul 2024 at 07:20, Tony Fabris @.***> wrote:

Also, @quog https://github.com/quog - attached to this message is a test script containing a prototype of the "fixed" version of the WebApiAuth function. Would you mind helping me out, and running this test file from the console and sending me its output? It should run from a Linux or Mac shell prompt, and it should be run from a computer on the same network as the NAS. You have to edit three variables at the top of the file to point to your server running DSM 7.x, and its username, rename the file from .txt to .sh, and chmod it to be executable. The output should look similar this, except yours should try to authenticate to version 7 instead of version 6:

DSM Authentication Test - Querying http://192.168.0.88:5000/webapi to see if it is DSM 6.x. DSM Authentication Test - Version Response: {"data":{"SYNO.API.Auth":{"maxVersion":6,"minVersion":1,"path":"auth.cgi"}},"success":true}. DSM Authentication Test - DSM version appears to be 6 - Result: {"data":{"SYNO.API.Auth":{"maxVersion":6,"minVersion":1,"path":"auth.cgi"}},"success":true}. DSM Authentication Test - Querying http://192.168.0.88:5000/webapi to see if it is DSM 7.x. DSM Authentication Test - Query did not succeed - Result: {"error":{"code":102},"success":false}. DSM Authentication Test - Attempting to authenticate to DSM version 6. DSM Authentication Test - Auth response: {"success":true}.

Script should be attached below. Thanks! DsmAuthTest.txt https://github.com/user-attachments/files/16257475/DsmAuthTest.txt

— Reply to this email directly, view it on GitHub https://github.com/tfabris/CrowCam/issues/77#issuecomment-2231973140, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5HTO3J3254F6XFEYZTUSDZMWTDLAVCNFSM6AAAAABK326XRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRHE3TGMJUGA . You are receiving this because you were mentioned.Message ID: @.***>

quog commented 1 month ago

Yes - worked fine.

Damian Douglas-Meyer 0414 419 624 @.***

On Thu, 18 Jul 2024 at 01:06, Tony Fabris @.***> wrote:

Also, @quog - Can you tell if logging to the Synology log (the function "LogMessage" in CrowCamHelperFunctions.sh) is working on DSM 7.x?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

quog commented 1 month ago

Hi Tony,

I understand re: the ping - I was following the precautionary principle of not running things as root that didn't need to be run as root.

My current version is working - I don't have time this afternoon to test the new version, but will try to over the weekend. See also my response about the test code.

Regards Damian Damian Douglas-Meyer 0414 419 624 @.***

On Thu, 18 Jul 2024 at 05:28, Tony Fabris @.***> wrote:

OK, I believe I've made the necessary changes to the master branch code to make it work under DSM 7.x, as long as it's run as "root".

I was able to do some limited testing on DSM 7.2 by logging into their online "demo" of DSM 7.2 from the Synology main web site. I see that the logging function still works, my tests to detect DSM6 or DSM7 are working, etc. I wasn't able to actually test whether the authentication API command works, though, because when you log into their online demo, you don't know the password for the user "demouser" and it terminates your session if you try to add new users or change the demouser password.

I haven't yet updated the "ping" command, because I have strong feelings about it needing to truly be a "ping" and not some other command. The trick of using nslookup instead of ping is a good trick, but it's not exactly the same thing. In particular, I'd like to have the option to change the configuration so that it pings some other server other than a DNS server to perform the network test. There could theoretically be situations where you would want that. Leaving it as a true ping gives it that flexibility.

@quog https://github.com/quog - If you are able, please try the new master branch version of the CrowCam code on your 7.2 server and tell me if it's working OK for you. The only important changed files are "CrowCam.sh" and "CrowCamHelperFunctions.sh", and you can leave your configuration files as they are. But you will need to reconfigure the tasks in Task Scheduler so that they run under the user "root" in order for the ping command to work.

— Reply to this email directly, view it on GitHub https://github.com/tfabris/CrowCam/issues/77#issuecomment-2234351630, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5HTO6SRP6FOKGEULISFBTZM3OXVAVCNFSM6AAAAABK326XRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZUGM2TCNRTGA . You are receiving this because you were mentioned.Message ID: @.***>

tfabris commented 1 month ago

@quog - Thanks so much for testing those things! I'm also stumped by why the authentication didn't work. As far as I can tell, the auth string in my code ends up being the same as the auth string in your (working) code and should therefore work.

If you run the new master branch version of the CrowCam code (keep your working code safe somewhere), if the auth string still fails, please contact me by email (tfabris at gmail) and we can work out what's wrong without spamming this GitHub thread.

tfabris commented 1 month ago

@quog - If the test code failed but the new Master branch code works, then one possibility could be that the password is obtained differently in my test code than in the main code. In the test code it's just a variable assignment at the top of the file:

password="myPassWord"

In the main code it reads the value from a different file on the disk:

read username password < "api-creds"

One possibility is that you simply mistyped the username and/or password in the test code, whereas it's correct in the "api-creds" file. A more likely possibility is that password contains some kind of special character that Bash interprets in a special way when performing a variable assignment, but the "read" command doesn't care about. Obviously don't tell me your password, but does the password contain something like a percent sign, a dollar sign, square or curly brackets, forward or back slashes, single or double quotes, or backticks? For example, the code password="pas$word" would definitely not work as intended. If that's the case, maybe it will work fine in the final code, and only fail in the test code.

quog commented 1 month ago

Hi Tony

The issue was special characters in the password. I changed the DsmAuthTest.sh to use read and it works.

bash DsmAuthTest.sh DSM Authentication Test - Querying http://127.0.0.1:5000/webapi to see if it is DSM 6.x. DSM Authentication Test - Version Response: {"data":{"SYNO.API.Auth":{"maxVersion":7,"minVersion":1,"path":"entry.cgi"}},"success":true}. DSM Authentication Test - DSM version appears to be 6 - Result: {"data":{"SYNO.API.Auth":{"maxVersion":7,"minVersion":1,"path":"entry.cgi"}},"success":true}. DSM Authentication Test - Querying http://127.0.0.1:5000/webapi to see if it is DSM 7.x. DSM Authentication Test - DSM version appears to be 7 - Result: {"data":{"SYNO.API.Auth":{"maxVersion":7,"minVersion":1,"path":"entry.cgi"}},"success":true}. DSM Authentication Test - Attempting to authenticate to DSM version 7. DSM Authentication Test - Auth response: {"data":{"did":"REDACTED","sid":"REDACTED"},"success":true}.

tfabris commented 1 month ago

@quog - That is fantastic, thank you so much. You’ve been a great help. Let me know if you encounter any other problems.

I’m going to leave this bug open until I can run DSM 7.x myself and double check a few things. Other than that, I think this bug should be fixed. (Crossing my fingers.)