Stream keys fail if they start with four base-10 digits

[tfabris]

The crowCam got into a situation today where YouTube showed that there was a scheduled stream waiting to start, but no stream was forthcoming, it just sat there in the Scheduled state all day. Logs looked like this:

2024-04-30 05:45:05 CrowCam Controller - Creating new YouTube Live Broadcast.
2024-04-30 05:45:20 CrowCam Controller - The streamStatus is not active. Value retrieved was: ready.
2024-04-30 05:46:19 CrowCam Controller - Local Synology stream key does not match YouTube stream name/key.
2024-04-30 05:46:19 CrowCam Controller - Local Key: 9470.
2024-04-30 05:46:19 CrowCam Controller - YouTube Key: 9470-rk**-****-****-****.
2024-04-30 05:46:19 CrowCam Controller - Updating local Synology stream name/key to be 9470-rk**-****-****-****.
2024-04-30 05:46:22 CrowCam Controller - The streamStatus is not active. Value retrieved was: ready.
2024-04-30 05:46:22 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
2024-04-30 05:46:22 CrowCam Controller - The lifeCycleStatus is not good. Value retrieved was: ready.
2024-04-30 05:46:22 CrowCam Controller - The recordingStatus is not good. Value retrieved was: notRecording.
2024-04-30 05:46:31 CrowCam Controller - The streamStatus is not active. Value retrieved was: ready.
2024-04-30 05:46:31 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
2024-04-30 05:46:31 CrowCam Controller - The lifeCycleStatus is not good. Value retrieved was: ready.
2024-04-30 05:46:31 CrowCam Controller - The recordingStatus is not good. Value retrieved was: notRecording.
2024-04-30 05:46:31 CrowCam Controller - Bouncing the YouTube stream for 4 seconds, because the stream was unexpectedly down during the quick test in the main code after having run the Test_Network function.

The above repeated all day.

After the first creation of the Live Broadcast, it did not try to create a new one, it just kept updating the key, but the key is wrong every time, local key is just the first four digits.

Investigating.

[tfabris]

Checked:

• Indeed the key in Surveillance Station is wrong, it's only the first four digits when I view it from the Surveillance Station Web UI by running the "Live Broadcast" widget.
• The Surveillance station wasn't updated last night, so it's not a new regression in Surveillance Station.
• I wonder if it's the fact that the key name contains the string "-r" and somehow that's being interpreted as a separate command?
• Looking at the terminal debug output is says that the key is being successfully set each time (Response: {"success":true}.)

I wonder if I need to quote-enclose or escape that key somehow?

NOTE: It's not the "-r" in the key, I tried it at the command line and even if I replaced the key with just numbers, it still failed, and got only the first four digits:

wget -qO- --load-cookies wgetcookies.txt --timeout=10 "http://localhost:5000/webapi/entry.cgi?api=SYNO.SurveillanceStation.YoutubeLive&method=Save&version=1&key=1234-1234-1234-1234-1234"

[tfabris]

Fails, gets only the first four digits:

• wget -qO- --load-cookies wgetcookies.txt --timeout=10 "http://localhost:5000/webapi/entry.cgi?api=SYNO.SurveillanceStation.YoutubeLive&method=Save&version=1&key=1234-1234-1234-1234-1234"

Works:

• wget -qO- --load-cookies wgetcookies.txt --timeout=10 "http://localhost:5000/webapi/entry.cgi?api=SYNO.SurveillanceStation.YoutubeLive&method=Save&version=1&key=abcd-abcd-abcd-abcd-abcd"
• wget -qO- --load-cookies wgetcookies.txt --timeout=10 "http://localhost:5000/webapi/entry.cgi?api=SYNO.SurveillanceStation.YoutubeLive&method=Save&version=1&key=efgh-efgh-efgh-efgh-efgh"

So it is something to do with the key, but I know not what yet, exactly. Some keys fail others succeed.

• Note: It's not the dashes. When I encode the dashes as "%2D" instead of the dash, it still gets the same behavior: Some keys work, others don't.
• Note: Quote-enclosing the key with single quotes causes the full key to be saved (in other words it sorta fixes the problem) but now the item saved in the LiveBroadcast widget includes the quotes themselves. The quotes themselves can't be part of the key name otherwise the stream fails. So there's gotta be another way around it.

Even URL-encoding every single digit in the string (including safe digits) like this, causes only the first four digits to be saved:

• wget -qO- --load-cookies wgetcookies.txt --timeout=10 "http://localhost:5000/webapi/entry.cgi?api=SYNO.SurveillanceStation.YoutubeLive&method=Save&version=1&key=%31%32%33%34%2D%31%32%33%34%2D%31%32%33%34%2D%31%32%33%34%2D%31%32%33%34"

[tfabris]

It's got something to do with the fact that the first group of the key is four base-10 digits. For instance, using the API to save this key fails:

• 3456-9470-9470-9470-9470

But saving this key works:

• 3a56-9470-9470-9470-9470

Note that the issue with saving the key only exists when using the API to save the key. The keys which fail only fail with the WGET api command that I'm using. If I paste the key directly into the Surveillance Station Web UI (into the Live Broadcast widget) then it works.

[tfabris]

I temporarily worked around the problem by hand-updating the key in Surveillance Station, and it worked and the stream started going again. Shortly thereafter, the system decided that it needed to split the stream at midday and created a new live broadcast with a new key, this time the key didn't have four pure decimal digits at the front, so it also worked as expected.

I don't know what this is?

• A bug in Wget where the digits of that parameter are somehow misinterpreted? Seems unlikely that it would be that first set of digits but not any of the others?
• A bug in Surveillance Station API which mis-interprets the input?

I think the latter is more likely, but it seems to me like the workaround will be the same regardless of the bug's root cause.

To do:

• Work around the issue by:
  • Detecting if the first four digits of the key are decimal numbers.
  • If so, then stop the stream and create a new live broadcast (which will automatically generate a new key).
  • Optional different, more generic ways to work around it:
    • Detect if the bouncing doesn't bring the stream back, then in that case create a new stream.
    • Detect if the local key doesn't match the youtube key even after updating it, then in that case create a new stream.

[tfabris]

I have done the following:

• Checkin bcd0859 detects if the local key doesn't match the youtube key even after updating it, and creates a new stream if that issue occurs.
• Tested that it works. I happened to be experiencing the problem this morning. It creates a new livestream. It has a side effect of having some additional weird errors in the log, and also leaving the "bad" livestream orphaned as an "upcoming" video, and you'll need to delete the orphan manually. But it records things correctly in the log and overall does the right thing.
• I have reported the bug to Synology Technical Support, ticket #3583111.

To do:

• Figure out how to clean up the orphaned video automatically (maybe it will get eventually cleaned up on its own without my intervention?).
• Observe this for a few days and see how frequently the issue recurs. I got two recurrences in the last two days, let's see if this continues to recur?
• Clean up the additional error messages which occur after the issue reproduces and the new video is created. Logs look like this:

  2024-05-01 12:36:37  CrowCam Controller - Updating local Synology stream name/key to be 6930-xb**-****-****-****
  2024-05-01 12:36:38  CrowCam Controller - Re-querying local stream key to ensure the update succeeded.
  2024-05-01 12:36:39  CrowCam Controller - Update failed: Local Synology stream key does not match YouTube stream name/key. Possible recurrence of GitHub issue #91. Creating new YouTube stream to work around the issue.
  2024-05-01 12:36:40  CrowCam Controller - Creating new YouTube Live Broadcast.
  2024-05-01 12:36:56  CrowCam Controller - New video is live. Video ID: 7R7_Flc1MNY Status: active Key: gbds-13**-****-****-****.
  2024-05-01 12:37:03  CrowCam Controller - The categoryId of the video UxIqdJJpXBnjaEy5yq1wZQ1714592202666342 titled CrowCam was - This does not match the desired categoryId of 15 - Fixing the categoryId now.
  2024-05-01 12:37:03  CrowCam Controller - Error setting categoryId to 15 on the video UxIqdJJpXBnjaEy5yq1wZQ1714592202666342. Output was { "error": { "code": 403, "message": "Forbidden", "errors": [ { "message": "Forbidden", "domain": "youtube.video", "reason": "forbidden" } ] }}.
  2024-05-01 12:37:07  CrowCam Controller - The video UxIqdJJpXBnjaEy5yq1wZQ1714592202666342 titled CrowCam was not found at the top of playlist PL8Fzg-YTf-GbfOHNxAAZEW9dixstuSztl titled CrowCam Archives - Instead the video 7R7_Flc1MNY was found in that slot - Fixing the playlist now.
  2024-05-01 12:37:07  CrowCam Controller - Error adding video UxIqdJJpXBnjaEy5yq1wZQ1714592202666342 to playlist id PL8Fzg-YTf-GbfOHNxAAZEW9dixstuSztl titled CrowCam Archives - Output was { "error": { "code": 404, "message": "Video not found.", "errors": [ { "message": "Video not found.", "domain": "youtube.playlistItem", "reason": "videoNotFound" } ] }}.
  2024-05-01 12:37:10  CrowCam Controller - Error uploading thumbnail to video UxIqdJJpXBnjaEy5yq1wZQ1714592202666342. Output was { "error": { "code": 500, "message": "Internal error encountered.", "errors": [ { "message": "Internal error encountered.", "domain": "global", "reason": "backendError" } ], "status": "INTERNAL" }}.

[tfabris]

I got a response from Synology support about my bugreport. They suggested a workaround of putting double quotes around the key, something I hadn't tried. I had already tried single quotes around the key, but that had only partially worked; the full key did save, but it also saved the single quotes into the key itself, thus making the key invalid. I hadn't thought to go forward and try double quotes around the key. Turns out that fixes it.

Though instead of doing double quotes in my script (since other parts of the code which touch that string are already using double quotes as other delimiters), I am using %22 instead, which also works successfully: &key=%221234-1234-1234-1234-1234%22

This should permanently fix the issue. Checkin 1e7904a adds the %22's around the key. I believe this is the true workaround to the bug so that it should never recur.

The prior checkin bcd0859, where it double-checks that the key got saved and creates a new stream if it didn't save correctly, that code is still in there as belt-and-suspenders protection if something else goes wrong with the key. But that code theoretically shouldn't get hit any more except in extreme emergencies. So I don't need to do my other "to do" items from above because the orphaned stream and the additional error messages shouldn't be hit regularly any more.