HLS video in Object Storage missing master streaming playlist and SHA256.json file

DeadSuperHero commented 1 month ago

Describe the current behavior

Our instance, Spectra.Video, has been on PeerTube for quite a long time. We originally made use of the legacy integration for Object Storage, and later upgraded to the official integration once a better solution came along.

I've been re-importing video files to existing PeerTube entries. Transcoding works, moving over to Object Storage works. However, some videos seem to get stuck in the Moving to Object Storage state, despite the job successfully completing. Playback still works on these videos, but the player can only serve files over HTTP rather than P2P. To be clear: this only happens every once in a while, and mostly seems to affect videos that either failed to move to Object Storage, or were originally being served through a legacy integration.

Here's an example video: https://spectra.video/w/8ZdnDTofjZ8wUvRQpNNZry

When playback is initially attempted, playback reports this error: HLS.js error: networkError - fatal: false - fragLoadError

There's also a failed XHR GET Request: XHR GET https://spectra.video/static/streaming-playlists/hls/40acfda8-17fd-4dab-9234-e844acc043be/0baceed4-96bc-400d-8fd6-47f71b37c9ab-1080-fragmented.mp4

That file doesn't exist anymore, since the video lives in Object Storage. When I check the Video Overview, the video files are correctly linked:

https://spectra-prod.us-east-1.linodeobjects.com/streaming-playlistshls/40acfda8-17fd-4dab-9234-e844acc043be/0baceed4-96bc-400d-8fd6-47f71b37c9ab-1080-fragmented.mp4
https://spectra-prod.us-east-1.linodeobjects.com/streaming-playlistshls/40acfda8-17fd-4dab-9234-e844acc043be/9191ddf7-0c9c-42f0-a769-0ea0b6d6d3d5-360-fragmented.mp4

Looking at my Object Storage folder for this particular video, I also noticed that there seems to be a few files missing? Compare and contrast:

The Affected Peertube Video

A Normal PeerTube Video

In our example video, these files appear to be missing in Object Storage:

...segments-sha256.json
...master.m3u8

Steps to reproduce

Have a video fail to move to Object Storage. Specifically, one that may no longer have a usable file locally on the server.
Attempt to either run an import job, or alter the video state and import a replacement video file.
In Video Overview, run "HLS Transcoding" on the affected video entry.
Once transcoding is complete, watch the "Move to Object Storage" job.

Describe the expected behavior

The transcoded HLS video and its required files should move over to Object Storage, and the video's state should be updated to 1. P2P playback should work, and the browser console shouldn't return errors about a failure to load fragments.

Additional information

PeerTube instance:
- URL: https://spectra.video
- Version: 6.1.0
- NodeJS version: v18.9.0
- Ffmpeg version: 6.1.1-3ubuntu5
Browser name, version and platforms on which you could reproduce the bug: Firefox 127.0.1, Ubuntu 24.04

Sentry issues:

DeadSuperHero commented 1 month ago

Update: I found the missing files in my local storage, and transferred them over to the corresponding Cloud Storage directory. The problem seems to persist, with the video entry still trying to refer to local storage?

DeadSuperHero commented 1 month ago

I feel like I'm close to fixing this. It seems like I need to update the entry in the videoStreamingPlaylisttable to point to the remote URLs in Object Storage?

I've managed to find the corresponding entry, now I just have to figure out how to set the playlistUrl field and the corresponding field for the sha256.json.

DeadSuperHero commented 1 month ago

Tried updating values in videoStreamingPlaylist, but the video player...still seems to be trying to refer to locally hosted files for playback?

Chocobozzz commented 1 month ago

HI,

Can you check in your logs/job info what is the status of the "move to object storage job" of this video? It's really weird the job doesn't complete/error.

Enabling debug logs in PeerTube server may help to check where the process is stuck when PeerTube tries to move the video on object storage

fabiopicchi commented 3 weeks ago

We have been through the exact same problem. After our huge import, we found that 666 videos are in the "To move to an external storage" state and 69 others are in "External storage move failed".

I still have to write a script to check their situation against certain possible scenarios, but I can confirm that after inspecting four of them manually, they all had their master playlist and segments json file missing from the object storage and the moving job was marked as completed. I could also find the files in my local filesystem.

As @DeadSuperHero figured out, you have to update both video and videoStreamingPlaylist tables in addition to upload the missing files to the object storage. I was able to put those four videos in a state that the HLS peer to peer transmission was working. Maybe you forgot to update some fields?

@Chocobozzz a funny thing about this situation. I tried rerunning the transcoding job for one of the failed videos and the exact same problem happened. The master file and the segments json were not uploaded and, hence, the videoStreamingPlaylist table was not updated. I didn't have any logs for that though :/

I'll see if I can generate some logs later today.

fabiopicchi commented 3 weeks ago

https://drive.google.com/file/d/1cC5UelNZqUDmcvVGTyn0L1Ep6FuJhZlj/view?usp=sharing

If you follow the video UUID "8dc4a48e-0a97-4cdd-bb3b-81242354c87b" in the file above you will see that it never runs the doAfterLastMove function, which is the one responsible for uploading the master playlist and the sha segments file.

fabiopicchi commented 3 weeks ago

I also developed this script to fix those interrupted uploads @DeadSuperHero

#!/bin/bash
set -euo pipefail

if [ -z "$1" ]; then
  echo "Usage: $0 <video_uuid>"
  exit 1
fi

VIDEO_UUID=$1

aws s3 sync "/var/www/peertube/storage/streaming-playlists/hls/private/${VIDEO_UUID}/" "s3://my-bucket/streaming-playlists/hls/${VIDEO_UUID}/"

SQL_SCRIPT=$(cat <<EOF
DO \$\$
DECLARE
    video_uuid UUID := '$VIDEO_UUID';
    playlist_filename VARCHAR(255);
    segments_sha256_filename VARCHAR(255);
BEGIN
    -- Update the video table state column
    UPDATE video
    SET state = 1
    WHERE uuid = video_uuid;

    -- Get the playlist filename and segments SHA256 filename
    SELECT "playlistFilename", "segmentsSha256Filename"
    INTO playlist_filename, segments_sha256_filename
    FROM "videoStreamingPlaylist"
    WHERE "videoId" = (SELECT id FROM video WHERE uuid = video_uuid);

    -- Update the videoStreamingPlaylist table
    UPDATE "videoStreamingPlaylist"
    SET
        "playlistUrl" = 'https://my.bucket.host/streaming-playlists/hls/' || video_uuid || '/' || playlist_filename,
        "segmentsSha256Url" = 'https://my.bucket.host/streaming-playlists/hls/' || video_uuid || '/' || segments_sha256_filename,
        storage = 1
    WHERE "videoId" = (SELECT id FROM video WHERE uuid = video_uuid);
END \$\$;
EOF
)

sudo -u postgres psql -d peertube_prod -c "$SQL_SCRIPT"

Chocobozzz commented 3 weeks ago

@fabiopicchi I think your issue is the same as https://github.com/Chocobozzz/PeerTube/issues/6459 and should be fixed by https://github.com/Chocobozzz/PeerTube/issues/6459#issuecomment-2259948978

@DeadSuperHero I saw that you fixed manually your videos in your blog post so I'm closing this issue. But please don't hesitate to comment when you notice de same bug in the future so we can fix the root cause of the issue :)

Chocobozzz / PeerTube