Stream stuck at "starting..." on 2019-06-23

tfabris commented 5 years ago

The morning of 2019-06-23 07:55 am, I observed the stream was stuck at "starting..." on the YouTube live dashboard page. The only stream data error being shown there was "Check Resolution" like it always says. No other errors were being shown.

Reolink app shows a perfectly fine live video feed at the correct 1296p/3mbps resolution setting.

YouTube stream settings, metadata, etc., are all correct. The secret key hasn't changed either.

Need to check Synology logs to see what's going on here.

Trying: Rebooting the camera from the reolink app.

tfabris commented 5 years ago

Rebooted camera at approximately 2019-06-23 8:05 am. Reolink app shows camera is back up, though the frame rate looks choppy. YouTube stream still stuck at "Starting..."

tfabris commented 5 years ago

Trying second reboot of camera at 8:09 am.

tfabris commented 5 years ago

Bouncing the camera didn't work. At least not at 8:09 am when I tried it twice.

I ran the script (remotely) and I have attached the output. The script thinks the stream is still up, so it's not trying to bounce it or anything. I have attached the script debug output, but the relevant part of the script thinks the stream is still up:

CrowCam Controller - streamStatus: active.
CrowCam Controller - healthStatus: ok.
CrowCam Controller - Live stream is up.
CrowCam Controller - Status - Network up: true - Stream up: true.

I waited until after the midday bounce. Still the stream was stuck at "Starting...." even after midday bounce time. My guess is that it would have bounced the stream since it's the height of summer. I checked the API quotas, those are fine. Nothing seems to be going weird with the YouTube API or anything. All of the stream metadata is correct and unchanged, including the stream live ID and the stream key and publicity status. Nothing is changed, all seems well except for the "Starting..." problem. The Comcast internet settings page (comcast quota) seems OK.

I have looked at the debug output and there is one way to tell that this is occurring. In the $liveBroadcastOutput there is "lifeCycleStatus": "liveStarting"

   "status": {
    "lifeCycleStatus": "liveStarting",
    "privacyStatus": "public",
    "recordingStatus": "recording"
   },

However, even if I detect this state, the most my script could do is bounce the stream. Well, we already did that, at midday, and the stream still isn't up.

I'm going to have to look at what this looks like while I'm at home. I wonder if it's a new Synology bug of some kind. Perhaps there's something going on with it.

Debug output (remote run).txt

tfabris commented 5 years ago

Returned home and looked at the system directly. Attached are some logs and screen shots.

The camera has been functioning normally as far as Synology is concerned. I look at the "Timeline" feature of the Synology Surveillance station for the down times, and it looks fine, all stored video looks normal.
Checking the camera settings in the "IP Camera" screen in Surveillance Station, everything looks correct and unchanged.
The Synology box has not needed to update itself lately. So this does not appear to be a side effect of updates. Surveillance station is at version 8.2.4-5847 which was released on 2019-06-11, weeks before this issue occurred, and hasn't updated since. Synology box itself shows no updates or reboots.
Rebooting the camera itself does not fix the issue. That happens automatically every day at 4:30 am and the logs show it successfully happening several times over the last several days. Still no fix.
Bouncing the Live Stream does not fix the issue. That has happened several times, both for midday bounces and for occasional ingestion errors. Still no fix.
Bouncing the DHCP router does not fix the issue. This happens nightly at 3:30 am and it never fixed the problem. Also tried manually while the stream was supposed to be up, still did not fix the issue.
Surveillance Station logs show nothing major wrong, however it did note a brief blip of a problem around 8:09 am on the day the issue started. Log entry was: "Information 2019/06/23 08:09:42 SYSTEM Live broadcast connection was lost because the status of camera [CrowCam] is abnormal." Only the once.
Synology system logs show that CrowCam Keep Alive knew something was wrong. If checking the API for "lifeCycleStatus": "liveStarting" is not enough, perhaps I should reinstate using YouTube-DL to check for upness checking.
Tried synoservicectl --restart pkgctl-SurveillanceStation and it did not fix the issue. Blast, I was counting on that one to work.
Note that doing --restart pkgctl-SurveillanceStation sat there for a while before returning to the shell prompt. I think that it truly waits there ("blocks") for the service restart to complete.
After bouncing the service, the Surveillance Station "Live Broadcast" feature screen briefly said "The selected camera is not operating under normal status" (screen shot attached). However that recovered after a minute or so and now read normal. However we still don't have a working stream.
Checking the camera settings in the "IP Camera" screen in Surveillance Station, everything looks correct and unchanged.
Once you get the camera working again, you need to check if "lifeCycleStatus": "liveStarting" in the API data changes to normal, and how quickly it changes to normal, so that once you find the fix, you can use "lifeCycleStatus": "liveStarting" as the way to detect the problem.
Trying a reboot of the Synology NAS 2019-06-26 9:42 am. Did not fix the issue.
Trying a reboot of my DHCP router/firewall 2019-06-26 9:53 am. Did not fix the issue.
Trying a reboot of my cable modem 2019-06-26 9:57 am. Did not fix the issue.
Double checked that the Surveillance station "Device Pack" was set to auto-update. Did not fix the issue.

I'm stumped so far.

System Log 2019-06-23 issue 42.txt

Surveillance Station Log 2019-06-23 issue 42.txt

Abnormal Camera Status Screen Shot issue 42

tfabris commented 5 years ago

Tried:

Turning off the DVR feature in the YouTube control panel, waiting a bit, then turning it back on. This caused the API to return "NoData" briefly, and made the script bounce the stream, but it still didn't fix the problem.

tfabris commented 5 years ago

Tried:

Turning off the DVR feature in the YouTube control panel for a long while. Didn't fix the issue.
Deleting all "upcoming" videos in the Beta youtube dashboard ala issue #40 . I deleted one too many, and seem to have deleted my main live stream in the process, which I now must re-create from scratch with a new key and new URL and everything.

Well, let's see if creating it from scratch again works.

tfabris commented 5 years ago

Creating the livestream from scratch again, worked.

So this must have been entirely a YouTube glitch. But how to detect and fix it?

Conclusions:

I tried everything that would easily be under the control of my script, short of creating a whole new livestream from scratch.
I don't think my script should create a whole new livestream from scratch, that's a bit too much. Its original goal was to maintain an existing livestream.

Detection of this issue:

After fixing, "lifeCycleStatus": "live" was in the LiveBroadcast output, so that could be a detection...
```
"status": {
"lifeCycleStatus": "live",
"privacyStatus": "public",
"recordingStatus": "recording"
},
```
However, since I don't have a programmatic way of FIXING it, I'm not sure I should do anything with that information at this time.

IMPORTANT NOTE:

Simply deleting the existing live stream and creating a new "Channel Default" live stream from scratch worked just fine. I didn't even need to change the script, or even stop the script, since the ingestion key is now automatically updated by the script.
The URL to the channel's default live stream link does not change when you make a new live stream. The random-numbers part of that link is channel's ID, so it doesn't change when you recreate a new live stream.
I could have fixed this whole thing remotely on day 1 if I'd simply had the confidence knowing that deleting and re-creating the live stream would Just Work even if I'm not at home to edit/stop/start the script.
Note that you have to fill in a lot of details of the live stream so that it fits your standards once you create the new live stream. Here are the settings I put in:
```
Title: CrowCam
```

Description: (detailed description that I keep in a separate text file)

Change Thumbnail: (I have a thumbnail that I keep on my local machine that I can upload when I need it)

Category: Pets & Animals

Stream Optimizations: Normal Latency.

(Enable DVR is on by default.)

IMPORTANT: Activate the "Make archive unlisted when complete" feature! This is unchecked by default.

Tags:

How to change the tags? Sometimes hard to find the screen.
Press ADVANCED SETTINGS
Press BASIC INFO
I know the above seems unintuitive but it worked for me.
If you take the string below with the commas, and paste the whole string in, with the commas, it works.

Crows, Squirrels, Animals, Live, Cam, Webcam, Birds, Birdwatching

tfabris commented 5 years ago

I've updated my repository with the notes on creating a new stream from scratch, and moved my title card into the repository. I should now be able to fix this issue remotely if I need to.

One idea I did not try, but now that I see what worked, I wonder if it would have worked:

The stream's secret name/key, on the YouTube site, has a "REFRESH" button next to it.
If I were to have pressed that button it would have regenerated the stream secret name/key while keeping all the other settings the same.
Then the script would have detected the key had changed, and done the needful to keep the stream going with the new key.
Would pressing that single button have fixed this issue?

Next time this happens, try that first instead of recreating the whole stream from scratch again.

tfabris / CrowCam

Stream stuck at "starting..." on 2019-06-23 #42