Player loops in old folder instead of detecting new folder

[scottveirs]

NOTE: I think this behavior occurs on live.orcasound.net, but this issue report is base on listening (via Macbook+Chrome) to beta.orcasound.net/orcasound-lab through one of the automated (cronjob) re-boots on the half hour.

I listened from ~10 minutes before the re-boot until about 20 minutes after the re-boot. The player got a 404 on segment live0329.ts one of the final 6 segments listed in the manifest but nonexistent in the associated data folder: streaming-orcasound-net/rpi_orcasound_lab/hls/1569799936. Then it seemed to cycle back some amount (30 seconds? 90?) and play again while appearing to download the now aging latest.txt and manifest.m3u8 files, and repeatedly getting the 404 error on live0329.ts .

I failed to check whether the newest latest.txt file contained the old timestamp (1569799936) or the new one (1569803534), but suspect the new timestamp is being downloaded, but the player is not responding to it -- due to a bug?

As of 18:24:00 local time (Seattle), the latest.txt file contains 1569803534 and I'll check it after the next re-boot in ~5 minutes...

[scottveirs]

Confirming this occurs on live.orcasound.net (as of 10/7/2020 on live-deploy branch with Rpi4 and boto-based orcanode branch), at least for Orcasound Lab feed. Player procures new latest.txt file with new Unix timestamp, but continues to load m3u8 file from previous timestamp.

Listening to the player in this situation, at the end of the last .ts segment, the sound is silent for ~5 seconds, then plays non-live sound for about 40 seconds (presumably the latest four 10s segments), then repeats silence/sound pattern.

[scottveirs]

Confirming via observation of the Orcasound Lab feed (with "no cache" checkbox checked in the Chrome developer console network tab) that the player loops over the final ~60 seconds. It seems to occasionally reload the latest 6-7 ten-second segments and gets a 404 on the final segment (live770) listed in the .m3u8 manifest (because live769.ts was the last segment to be uploaded, presumably the generation and/or upload of segment 770 was interrupted by the container going down per the 12:30 cron job).

[scottveirs]

Flagging some user feedback from "pro" listeners that is likely related to this issue... Note the drop in performance seems to occur in multiple browsers, but are exclusively related to Apple devices:

This may be an edge case, and it could mostly resolve when we move away from such frequent streaming container restarts (currently every 6 hours). These users are probably listening for many hours at a time on many days, so are much more likely to encounter intermittent performance issues (e.g. when a container restarts on the node to which they are listening).

[scottveirs]

Today listening to Orcasound Lab for most of the day, I experienced the same behavior (in Chrome on OSX) as documented previously, here -- https://github.com/orcasound/orcasite/issues/28#issuecomment-704757040

[paulcretu]

Looping / exponential backoff is completely removed in v3, so I don't expect that to be a problem anymore (it was confusing anyway). Now the player should display an error icon/message like this:

If a node is offline, it's possible the player will play the last few seconds before it went offline before either showing the error icon or a loading spinner, but in either case, shouldn't loop.

What may still be an issue is switching to a new timestamp after a restart. I haven't tested it yet, so I'll leave this issue open until we see how v3 behaves across restarts.

I thought one potential cause could be improper cache headers on the timestamp.txt file, but if it's loading the new one then something else must be going wrong (still would be good to double check the cache headers, that's an orcanode issue).

[paulcretu]

@scottveirs have you observed any issues with the player switching streams between node restarts? When are the nodes currently configured to restart?