Closed DeadSuperHero closed 3 weeks ago
Update: I found the missing files in my local storage, and transferred them over to the corresponding Cloud Storage directory. The problem seems to persist, with the video entry still trying to refer to local storage?
I feel like I'm close to fixing this. It seems like I need to update the entry in the videoStreamingPlaylist
table to point to the remote URLs in Object Storage?
I've managed to find the corresponding entry, now I just have to figure out how to set the playlistUrl
field and the corresponding field for the sha256.json.
Tried updating values in videoStreamingPlaylist
, but the video player...still seems to be trying to refer to locally hosted files for playback?
HI,
Can you check in your logs/job info what is the status of the "move to object storage job" of this video? It's really weird the job doesn't complete/error.
Enabling debug logs in PeerTube server may help to check where the process is stuck when PeerTube tries to move the video on object storage
We have been through the exact same problem. After our huge import, we found that 666 videos are in the "To move to an external storage" state and 69 others are in "External storage move failed".
I still have to write a script to check their situation against certain possible scenarios, but I can confirm that after inspecting four of them manually, they all had their master playlist and segments json file missing from the object storage and the moving job was marked as completed. I could also find the files in my local filesystem.
As @DeadSuperHero figured out, you have to update both video and videoStreamingPlaylist tables in addition to upload the missing files to the object storage. I was able to put those four videos in a state that the HLS peer to peer transmission was working. Maybe you forgot to update some fields?
@Chocobozzz a funny thing about this situation. I tried rerunning the transcoding job for one of the failed videos and the exact same problem happened. The master file and the segments json were not uploaded and, hence, the videoStreamingPlaylist table was not updated. I didn't have any logs for that though :/
I'll see if I can generate some logs later today.
https://drive.google.com/file/d/1cC5UelNZqUDmcvVGTyn0L1Ep6FuJhZlj/view?usp=sharing
If you follow the video UUID "8dc4a48e-0a97-4cdd-bb3b-81242354c87b" in the file above you will see that it never runs the doAfterLastMove
function, which is the one responsible for uploading the master playlist and the sha segments file.
I also developed this script to fix those interrupted uploads @DeadSuperHero
#!/bin/bash
set -euo pipefail
if [ -z "$1" ]; then
echo "Usage: $0 <video_uuid>"
exit 1
fi
VIDEO_UUID=$1
aws s3 sync "/var/www/peertube/storage/streaming-playlists/hls/private/${VIDEO_UUID}/" "s3://my-bucket/streaming-playlists/hls/${VIDEO_UUID}/"
SQL_SCRIPT=$(cat <<EOF
DO \$\$
DECLARE
video_uuid UUID := '$VIDEO_UUID';
playlist_filename VARCHAR(255);
segments_sha256_filename VARCHAR(255);
BEGIN
-- Update the video table state column
UPDATE video
SET state = 1
WHERE uuid = video_uuid;
-- Get the playlist filename and segments SHA256 filename
SELECT "playlistFilename", "segmentsSha256Filename"
INTO playlist_filename, segments_sha256_filename
FROM "videoStreamingPlaylist"
WHERE "videoId" = (SELECT id FROM video WHERE uuid = video_uuid);
-- Update the videoStreamingPlaylist table
UPDATE "videoStreamingPlaylist"
SET
"playlistUrl" = 'https://my.bucket.host/streaming-playlists/hls/' || video_uuid || '/' || playlist_filename,
"segmentsSha256Url" = 'https://my.bucket.host/streaming-playlists/hls/' || video_uuid || '/' || segments_sha256_filename,
storage = 1
WHERE "videoId" = (SELECT id FROM video WHERE uuid = video_uuid);
END \$\$;
EOF
)
sudo -u postgres psql -d peertube_prod -c "$SQL_SCRIPT"
@fabiopicchi I think your issue is the same as https://github.com/Chocobozzz/PeerTube/issues/6459 and should be fixed by https://github.com/Chocobozzz/PeerTube/issues/6459#issuecomment-2259948978
@DeadSuperHero I saw that you fixed manually your videos in your blog post so I'm closing this issue. But please don't hesitate to comment when you notice de same bug in the future so we can fix the root cause of the issue :)
Describe the current behavior
Our instance, Spectra.Video, has been on PeerTube for quite a long time. We originally made use of the legacy integration for Object Storage, and later upgraded to the official integration once a better solution came along.
I've been re-importing video files to existing PeerTube entries. Transcoding works, moving over to Object Storage works. However, some videos seem to get stuck in the
Moving to Object Storage
state, despite the job successfully completing. Playback still works on these videos, but the player can only serve files over HTTP rather than P2P. To be clear: this only happens every once in a while, and mostly seems to affect videos that either failed to move to Object Storage, or were originally being served through a legacy integration.Here's an example video: https://spectra.video/w/8ZdnDTofjZ8wUvRQpNNZry
When playback is initially attempted, playback reports this error:
HLS.js error: networkError - fatal: false - fragLoadError
There's also a failed XHR GET Request:
XHR GET https://spectra.video/static/streaming-playlists/hls/40acfda8-17fd-4dab-9234-e844acc043be/0baceed4-96bc-400d-8fd6-47f71b37c9ab-1080-fragmented.mp4
That file doesn't exist anymore, since the video lives in Object Storage. When I check the Video Overview, the video files are correctly linked:
https://spectra-prod.us-east-1.linodeobjects.com/streaming-playlistshls/40acfda8-17fd-4dab-9234-e844acc043be/0baceed4-96bc-400d-8fd6-47f71b37c9ab-1080-fragmented.mp4
https://spectra-prod.us-east-1.linodeobjects.com/streaming-playlistshls/40acfda8-17fd-4dab-9234-e844acc043be/9191ddf7-0c9c-42f0-a769-0ea0b6d6d3d5-360-fragmented.mp4
Looking at my Object Storage folder for this particular video, I also noticed that there seems to be a few files missing? Compare and contrast:
The Affected Peertube Video![image](https://github.com/user-attachments/assets/0dd21d43-3474-4936-9ad2-67420f423562)
A Normal PeerTube Video![image](https://github.com/user-attachments/assets/4eccdac9-fa03-47af-915c-af0f33f045fe)
In our example video, these files appear to be missing in Object Storage:
Steps to reproduce
Describe the expected behavior
The transcoded HLS video and its required files should move over to Object Storage, and the video's state should be updated to
1
. P2P playback should work, and the browser console shouldn't return errors about a failure to load fragments.Additional information
PeerTube instance:
Browser name, version and platforms on which you could reproduce the bug: Firefox 127.0.1, Ubuntu 24.04
Sentry issues: