Rewrite code to work around YouTube deprecating persistent/default stream.

[tfabris]

The CrowCam code depended on the YouTube features "Default Stream" and "Persistent Stream", which they deprecated on approximately 2020-09-10. The code no longer works correctly.

To do:

• Rewrite the code to do the necessary steps to create a new broadcast, and bind the stream to that broadcast, each time it is required.
• In the process, also move the CrowCam to a new YouTube channel so that we can separate the CrowCam from the Vixy & Tony YouTube channel.
• Update the CrowCam Cleanup code, and also my side project "web site update" code (which maintains the VixyAndTony/CrowCam.html page data), so that it handles the new channel. For the old archived videos in the channel that I'm keeping, investigate if it is easier to just keep them there and have those programs maintain both channel archives, or copy all the videos from the old archive to the new one (if I do it that way, I would need to rebuild all of the bit.ly links embedded in the video descriptions).

Helpful links:

• Breaking change notification: https://developers.google.com/youtube/v3/live/revision_history#release_notes_04_16_2020
• More detailed information about IsDefaultBroadcast, etc.: https://developers.google.com/youtube/v3/live/docs/liveBroadcasts#snippet.isDefaultBroadcast
• Life of a broadcast guide: https://developers.google.com/youtube/v3/live/life-of-a-broadcast
• Migration guide: https://developers.google.com/youtube/v3/live/guides/migration-guide-default-broadcasts

[caso]

First of all, big thank you for the huge work you have done with this! These scripts have been a big help with my own bird box setup. Anyway, do you have plans to continue working with this 'new broadcast and bind the stream to that broadcast' routine?

[tfabris]

I'm so glad that the script was helpful to you in the past.

I want to work on it, but I'm not sure when I'll get back to it. There's a secondary factor at work here:

• Around the same time that YouTube deprecated those "persistent stream" features, we had to replace our cable modem with a newer model, due to problems with the old one. The new model of cable modem is more compatible with the cable company's features, which solved the problems but also now the cable company can successfully track our monthly usage, when they didn't before. I think that the monthly bandwidth tracking must be built into the cable modem firmware, and the old modem didn't have that feature.
• The CrowCam, if we stream it to YouTube, eats up a the majority of our monthly bandwidth stream allotment, leaving very little bandwidth left for things like movie streams and Zoom calls.
• This is a problem, especially since all of our household is working-from-home now, and our internet connection is important. In order to keep us from busting our monthly cable company quota, we can't stream the CrowCam to YouTube any more.
• Since the script no longer meets my personal needs (since we no longer maintain the YouTube stream), I'm not as motivated to work on the script. Also, since I'm not using it every day like I did before, I'm not there to notice bugs and changes in its behavior, so I'm not able to test and maintain it.

But... The beauty of GitHub is that I should theoretically be able to accept code changes from other people and incorporate it into this script. If you'd like to give it a shot and see if you can get it working under the new YouTube paradigm, I'd love to work with you on that.

[dkelson]

@tfabris If you could identify and outline the changes that need to be made, I could implement them and then maintain.

[tfabris]

@tfabris If you could identify and outline the changes that need to be made, I could implement them and then maintain.

@dkelson - Thanks so much for the offer. All the information I have is at the top of this thread. It’s a little vague because I don’t have all the knowledge myself. If I knew exactly which code changes to make, I would have already made them.

There are some things there which are just for me, not possible for others to do: for example, moving my personal video feed to a new YouTube channel and updating my personal website code. Clearly you couldn't do those things for me.

The first item on the todo list, that is the one that could be taken on by someone else, and submitted as a pull request: “Rewrite the code to do the necessary steps to create a new broadcast, and bind the stream to that broadcast, each time it is required.” This will require that you analyze and understand the current code behavior, and understand why YouTube’s API changes broke it. There are some links included there which should helpful in understanding why it broke. The code depended on the stream being the same stream all the time, and just needing a slight poke once in a while to keep it going. Now YouTube has a whole new set of requirements about setting up a new stream from scratch each time you want to restart the video playback.

If you want to take this on, I can try to support you where needed. You’ll need your own private YouTube channel, Synology, and security camera to run it on. Once you’ve got something that works, you can do a PR and we’ll review it together.

Let me know if you’re up for it!

[dkelson]

@tfabris "You’ll need your own private YouTube channel, Synology, and security camera to run it on"

I have these three things.

This I need to work on wrapping my mind around. Since I'm not the original author, it's a more of an uphill climb but doable. "analyze and understand the current code behavior, and understand why YouTube’s API changes broke it."

[tfabris]

@dkelson ,

Indeed, yes, learning the code flow of someone else's code is an uphill climb. But that's always the case when one takes on this kind of a task. It's what happens in all software development (open-source or otherwise), and I'd go so far as to say that it's the hardest aspect of software development. I've never seen anyone teaching a class on how to do it.

On the good side, my code is heavily commented with as much information as I could possibly include. And I'm also available to answer questions: if you choose to accept this mission, feel free to contact me by private email once you start running into things that you don't understand. My email address is at the Vixy & Tony web site.

Thanks!

[tfabris]

I have switched to an ISP which doesn't have a data cap. So this script could theoretically be made to work again. Currently investigating whether or not I can do this.

Issues which are currently presenting themselves (more to come):

• Deprecation of Default Broadcast and Persistent stream features as described above.
• Deprecation of the features I was using for getting an OAuth token to communicate with the YouTube API.
  • Description of the change: https://developers.google.com/identity/protocols/oauth2/resources/oob-migration
  • Variables affected:
  • refreshToken
  • accessToken
  • Functions affected:
  • CrowCamHelperFunctions.sh
    • YouTubeApiAuth()
    • GetRealTimes()
  • CrowCam.sh
    • Test_Stream()
    • Fixing the Stream Key
    • Fixing the Privacy Status
    • Bouncing the stream
  • CrowCamCleanup.sh
    • (all)
  • CrowCamCleanupPreparation.sh
    • (all)

[tfabris]

Status so far:

• Note: Doing most of this work on the issue69 branch, some of these changes aren't in Master yet. Most recent checkin on issue69 branch is 01a112e and contains all of the following:
• OAuth code retrieval seems to work now.
• OAuth code retrieval has a somewhat onerous extra step now. The google authentication process won't give you the auth key themselves in a popup box any more (that's what they called the OOB flow) - It has to call to your app's web page and feed it the code instead. You can work around this by using "http://localhost/somewebpage.html" as your redirect value, and running a web server on localhost that displays the code parameter yourself. (UPDATE: Fixed in 595b948 - The user can now just copy the URL from the URL bar even if no localhost page is running.)
• I have confirmed that CrowCamCleanup.sh is more or less functional now with the new OAuth steps.
• I have copied all the old archived CrowCam videos from my old YouTube channel to my new YouTube channel.
• I did the move by adding some new code to "TestFile.sh" which shows how to accomplish that.
• I significantly sped up the CrowCamCleanup.sh by changing the way it parsed the JSON (same technique I used in that new code I added to "TestFile.sh". Details are in issue #72 - there is still more speedup which can be done.

TO DO:

• Need to write proper instructions for doing that crazy thing with the localhost web page, described above. (UPDATE: Done in 595b948 )
• Need to finish fixing Issue #69 by updating CrowCam.sh so that it can correctly rebuild the stream each time it needs to be restarted.
• Need to fix issue #73

[tfabris]

Current status:

• Some fixes are in for issue #69.
• The code branch is now moved to Master.
• I'm leaving this bug open for a little while while I do some testing to feel whether or not it's solid.
• There are some new bugs which have come out of the gas leak years (2020-2023) which I still need to fix, I've made them each their own bugs:
  • Issue #75
  • Issue #73
  • Issue #72
• There is also a problem where CrowCamKeepAlive is not working properly at the moment. It needs an overhaul. I've put that into this issue:
  • Issue #70

To do:

• See how CrowCam does, running on our camera system, over the next several days. If it seems good, then we can close this bug

[tfabris]

There appears to still be something wrong here.

Yesterday there should have been three segments, the last segment running from about 7pm to about 8pm (since the prior segment runnng up to 7pm was four hours long and got split at the four hour mark). The Synology log showed the four hour split mark as normal, but that last third segment never showed up on the list of videos for the channel. Not in the Uploads playlist nor the CrowCam Archives playlist. I don't know if there was even a stream up at that point or not.

Then the following morning, there was a similar situation with the 6:35am stream at sunrise. The log said it was starting the stream but nothing was showing up on the youtube channel.

The CrowCam.sh is not throwing any errors or other messages to the Synology log, it just had the 6:35am stream startup message and that's it.

I went to YouTube's Live Dashboard stream at https://www.youtube.com/live_dashboard and it showed that it was expecting some stream data to reach it. There were no errors or anything, it just had the spinny icon saying that I needed to connect my streaming software to send data. But looking at the Synology "Live Broadcast" feature, it was on (and the log said it had been turned on at 6:35am), and the stream key was correct (the Synology one matched the one in the YouTube Live Dashboard).

I'm going to investigate by running the script on the Synology and looking at its shell debug output.

[tfabris]

There are no error messages when running the CrowCam.sh script at the shell prompt. It sees that the Synology "live broadcast" feature is on as it should be, it retrieves the bound stream ID correctly, successfully checks that the stream key is correct, makes sure there are no network glitches, and says everything is just hunky-dory, no need to bounce the stream.

From the Synology perspective, everything is fine.

TO DO:

• see if there is something I can query of the bound stream, see if there is a field that tells me whether a stream is present on the channel yet or not.

[tfabris]

The code already looks for things like "videoIngestionStarved" but that's not happening in this case, the YouTube API is not returning that particular error or any other error, even though that's exactly the situation it's encountering (there seems to be nothing sending video data to it at the moment).

I have noticed that when I use the API to query the live stream, I see this response (notice how it says healthStatus is Good even though there is no actual stream to be healthy, wow that's fscked up):

{
  "kind": "youtube#liveStreamListResponse",
  "etag": "asdf1234asdf1234asdf1234asdf1234",
  "pageInfo": {
    "totalResults": 0,
    "resultsPerPage": 5
  },
  "items": [
    {
      "kind": "youtube#liveStream",
      "etag": "asdf1234asdf1234asdf1234asdf1234",
      "id": "asdf1234asdf1234asdf1234asdf1234asdf1234",
      "status": {
        "streamStatus": "active",
        "healthStatus": {
          "status": "good"
        }
      }
    }
  ]
}

However according to the API documentation of the LiveStreams Resource, there is more data in other fields if I query for them, for example, the stream resolution and frame rate is in the "CDN" field. Trying this: Bounce the Synology live broadcast, see if it makes the stream come back, and if it does, see if any of those other things can be a good coalmine canary (since the YouTube API is lying in its "status" field).

[tfabris]

Nope, the LiveStreams resource seems identical. Even when I query for everything, id, snippet, cdn, and status, it has the same output. So the LiveStreams resource is not the correct resource to look for.

However if you query the LiveBroadcasts resource, I do see what I hope is a good query. Not certain about this one but we'll see.

• If you query the LiveBroadcasts resource with the "mine=true" parameter
• And you query for the part "status"
• AND YOU LOOK AT ONLY THE FIRST RESULT because there may be older broadcasts in the list...
• You will see this:

    "status": {
      "lifeCycleStatus": "live",
      "privacyStatus": "public",
      "recordingStatus": "recording",
      "madeForKids": false,
      "selfDeclaredMadeForKids": false
    },

When I was reproducing the problem I could see that the values:

• lifeCycleStatus was something other than "live" (such as "complete" or "ready")
• recordingStatus was something other than "recording" (such as "notRecording")

To do: Update the Test_Stream function to also check for these, and bounce the stream if those things aren't what is expected.

Note: There is also a tantalizing value "boundStreamLastUpdateTimeMs" in the "contentDetails" part...

      "contentDetails": {
        "boundStreamId": "UxIqdJJpXBnjaEy5yq1wZQ1705457117285100",
        "boundStreamLastUpdateTimeMs": "2024-03-18T18:39:32Z",
        "monitorStream": {

However this is not a useful value to query. It does not change, even when the stream is live and working. This is just the moment that the ID changed, not the moment of the video pixels were updated.

[tfabris]

The detection of lifeCycleStatus and recordingStatus is a good indicator. However, if lifeCycleStatus is "complete" then we have to figure out how to start a new video and go live. I tried doing the simplest thing which is to perform an INSERT operation:

  'https://youtube.googleapis.com/youtube/v3/liveBroadcasts?part=snippet%2Cstatus&key=[YOUR_API_KEY]' \
  --header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data '{"snippet":{"title":"CrowCam","scheduledStartTime":"2024-03-18T20:55:00Z"},"status":{"privacyStatus":"public"}}' \
  --compressed

This didn't quite work the way I'd hoped. It created an "UPCOMING" video which I now have to figure out how to transition to being live. Investigating...

Aha, OK we do have to do the whole rigamarole described in the lifecycle of a broadcast document. Working on that.

[tfabris]

With d7fa902, a new stream is finally created at sunrise and when the stream is being split.

To do:

• Detect situations where a brief bounce of the stream doesn't immediately fix the problem, and just create a new stream in those situations instead of trying to re-bounce again.

[tfabris]

I believe that 5f5280e - Final fix for issue #69 ? is the final fix for issue #69.

This morning I awoke to a situation where, despite there being nothing else that had gone wrong with the stream, it was down, and the stream was in the state recordingStatus: recorded lifeCycleStatus: complete.

That looks like a really good litmus test for the need to rebuild the livestream. So I added code to do precisely that, at the place where that detection occurs. When I put it into production, it worked exactly as expected immediately. Does that mean it works? (You shut up, Betteridge.)

I had originally thought that I would need to do a fancier check, trying a short bounce first and if that doesn't work then try a long bounce. Luckily the log messages about recordingStatus and lifeCycleStatus showed me that there was a perfectly good way to check it directly without having to do anything fancy on my side.

Closing this bug finally! If anything else comes up, it will probably need to be a separate bug anyway.

PS: @caso and @dkelson - Feel free to try the code in its current state. I'm still working on some existing issues, so keep a lookout for new code updates in the near future.