Stream name, description, and settings occasionally get clobbered, killing the stream.

[tfabris]

This is a similar issue to bug #24 - but it's not the same data field.

On 2019-06-02, there was a problem:

• The CrowCam stream went down.
• I noticed network problems in the afternoon on The Agora network as follows:
  • Slow upload of videogame savegames to playstation network.
  • Nest Thermostat lost connection to the WLAN.
• Around the same time I heard that Google Cloud Services had had a major outage that day for several hours.
• The CrowCam stream had went down for the afternoon and I left the house without knowing that it had done so.
• Checking later that night, I see that the Stream Key is unchanged, but all the video metadata, such as the title and description, the thumbnail, and the public/unlisted setting, were all wiped and were set back to the channel defaults.
• Not sure what is up with the stream remaining down though. Looked at my code, and the code says that I should not have prevented a proper stream-uppage due to the title difference. Why did the stream stop?
• After fixing the title, description, and other metadata, the stream resumed in the morning. Though it wasn't publicly visible because I had accidentally set the stream to "unlisted". Note to self, remember, the stream is public, but you configure the settings so that archives are unlisted.

This is similar to the situations which occur when the stream key gets clobbered out from under me. However it's different because the key didn't change, but the title and metadata did change. So what happened here?

To do:

• When you get back home, look at the logs and save them off and see what happened.
• Consider whether you can figure out how to fix this situation automatically when it recurs.

[tfabris]

I analyzed the logs (attached). Conclusions below.

Likely root cause (TLDR):

• Google Cloud Services outage that particular afternoon, reported in the news.
• Script behaved as expected. Need to decide what to do about issues of the API returning system errors instead of stream errors.

Unrelated:

• Nest thermostat's problem was unrelated. It was due to the WiFi chip in the thermostat going bad.

Details of what was found in the logs and video archives:

• Video for this day was this video: https://youtu.be/liFxs4GRrcM
• 2019-06-02 11:54 am. - Bounced due to ""code": 503, "message": "Backend Error""
• 2019-06-02 12:05 pm. - Bounced due to "The streamStatus is not active. Value retrieved was: inactive" and "The healthStatus is not good. Value retrieved was: noData." and ""code": 500, "message": "Backend Error""
• Indeed the stream for that time range "jumped" from 11:54 to 12:08 on the video archive, consistent with the bounces in the logs.
• However there is a problem that the stream archive did not "split" at the bounce point as expected, despite my longer 105-second split-time already being programmed in there. I wonder if the failure to split the stream was due to the YouTube Google Cloud Hosting services having backend errors?
• Due to the script recording that a bounce occurred around 11:54 am, the algorithm decided that a midday bounce would not be needed. So it just went along and kept going until the stream stopped entirely at 1:43pm (more on this below).
• The stream worked fine most of the time up until 1:43pm, despite the logs being full of a bunch of YouTube API errors and Backend errors. The hysteresis worked and kept finding that the API came back up all of those times from 12:07 to 1:43. They all say "Live stream came back up." This must have been during the time that Google Cloud Services were having the worst of their outage that day.
• At 1:43pm it finally busted the hysteresis limit and bounced the stream.
• There was a "inactive/noData" error at 1:51pm after the stream bounce, and it bounced the stream once more.
• The logs show that the stream was back up and working after 1:55pm, mixed in with a ton of "Backend Errors" continuing. But I have no archive of that stream. I see "Live stream came back up" at 2:07 pm and 2:28 pm, and then after 2:28pm, no more errors came from the Google API. But I have no video archive of any of that time after 1:43 pm.
• My guess is that the stream was working, but the saved files for those times were lost during the Google Cloud Services outages on that day. Perhaps the files are lost due to the clobbering of the name and metadata of the stream.

Logs attached below.

2019-06-02 and 03 - CrowCam Logs.txt

[tfabris]

To do:

• Add some more logging to display the stream's title and description, which I think is included in the API responses. Debug log them at all times, and if they don't match "CrowCam", log them in "err" mode.
• Consider updating the title and metadata if they change. If so, also include all the other metadata updates in the code as well: Keywords, latency settings, making saved archives unlisted. Look at the possibility of doing that. Right now you don't update the stream key (as a "safety") if the name doesn't match, so that's a catch-22. Do you really need the safety or can this all just be blindly written to the channel as long as it's the right default channel?

Also:

• Consider editing the API code so that certain errors don't trigger a "stream is down" problem. For instance, times when the API responses are blank, or they say "backend error" or whatnot. Maybe I should not fail the stream if it says any of those things. I think it over-bounced when Google Cloud Services were having trouble and I was getting those backend error.

[tfabris]

Done:

• The most recent checkin addresses the issue bouncing the stream when the YouTube API is failing.
• The stream key title is already logged to synology if it doesn't match, and the entire time that this issue #37 was occurring, looking at the logs, all of the YouTube API calls which attempted to access that title were failing, so I was getting a blank title field each time.

Conclusion:

• When the issue occurred, it was simultaneous with the downtime of the Google Cloud Services and the YouTube API. The very API calls that I would have needed to fix the stream title and metadata were not working at all. Even after the services seemed to recover themselves later in the day, nothing in the log shows that the API was successfully reaching the title field of the video stream.
• Therefore even if I could figure out the code to fix the title and metadata automatically, that could would not have solved this problem.

To do:

• Update production server with the latest checkin.
• Close bug as "fixed" for the bad-api over-bouncing issue.
• Close bug as "can't fix" for the title/description getting clobbered.

[tfabris]

• Production server is updated with the latest checkin, it's survived a few days and looks like it's running OK.
• Since implementing the bad-api over-bouncing issue, the google API services have not repeated the original problem, so I cannot be certain that the changes will work correctly. However I tested them pretty good with fake values so we can just leave that in place and call it good.
• Closing bug as "fixed" for the bad-api over-bouncing issue.
• Closing bug as "can't fix" for the title/description getting clobbered.

[tfabris]

Issue recurred on the morning of 2019-06-14, but this time, the Google API services weren't down. The stream got changed from "public" to "unlisted" without me doing anything to it. Luckily no other settings got changed, but that setting essentially made the stream go down.

There were no errors in the logs other than a "videoIngestionStarved" blip.

To do:

• Code should check all the important settings and fix them.

[tfabris]

Issue recurred on the morning of 2019-06-18, the stream got changed from "public" to "unlisted" without me doing anything to it. No other settings got changed. My current work-in-progress code successfully detected this status and reported it.

At the current time I'm still trying to figure out how to have that code automatically fix the issue. Right now the code only succeeds at creating a new "upcoming" stream with the desired settings rather than editing the settings of the current live broadcast.

[tfabris]

Issue recurred on the morning of 2019-07-04, the stream got changed from "public" to "unlisted" without me doing anything to it. No other settings got changed.

[tfabris]

I figured out what went wrong with the code which was trying to fix the privacyStatus. It was a minor syntax error in the Curl command line parameters, which caused it to be a POST instead of a PUT, thus making the API command be an "insert" rather than an "update", thus creating a new upcoming livestream rather than editing the current livestream. Now that it's fixed, I can update any value in the livestream that I want.

With that fixed, I now have a choice:

• Fix only the privacyStatus if it's gone bad. The desired privacyStatus is stored in a variable in the crowcam-config file.
• Fix all of the values in the entire live stream, including the video description, the tags, the thumbnail picture, etc.

Currently I am deliberately only fixing privacyStatus, not any of the other values.

Reasoning:

• The privacyStatus value has proven repeatedly that it is a problem child. It gets clobbered frequently, without any of the other stream details getting clobbered. This is the critical item to fix, and I've now fixed it.
• If I were to fix any of the the other values, I'd need to be saving their desired states in the script config file as well.

The problem with having the other values in the config file is that it removes my freedom and flexibility. For instance, I would have to put the entire video description, tags, etc, into the configuration file of the script. Then each time the script had to fix a clobber, those things would all be overwritten with the config file's versions of those values. This means that I would lose my freedom to edit those things from the YouTube user interface: Any edits I make would get overwritten any time they're clobbered. Any time I feel like updating the video description, I'd have to edit the code to match and then put that code into production on the Synology.

As it stands right now, I've saved that information in a file here, so that I can remotely fix serious clobbers if I have to, like the ones that prompted issues #37 and #42. So any time I update the video description, I also update that file as well, but it's not required to update any code files. I still have freedom right now.

(Also: I haven't figured out how to update the thumbnail picture correctly yet, and it looks complicated.)

So, for right now: fix privacyStatus with code, but only fix the rest of the clobbered stuff by hand.