create-move-video-storage-job fails to download video

normen commented 1 week ago

Describe the current behavior

Currently, when trying to use create-move-video-storage-job to download a S3 stored video the video doesn't get downloaded completely and is stuck in limbo. The video page shows "This video is being copied to the server storage and might not work properly" (translated that from german), the copy job shows as "completed" in the job list and there is no errors in the log (INFO level, see below).

The file situation after this is as follows, it seems like not all files did transfer:

On S3:

video-segments-sha256.json
video-master.m3u8

Locally:

video-1080-fragmented.mp4
video-1080.m3u8

Steps to reproduce

Create Peertube with S3
use create-move-video-storage-job
See failed video

Describe the expected behavior

The video should download and be used locally

Additional information

PeerTube instance:
- URL: https://tube.bitwaves.de
- Version: 6.2.1
- NodeJS version: docker
- Ffmpeg version: docker
- S3 Provider: Strato HiDrive
Server Log:

[tube.bitwaves.de:443] 2024-09-06 08:58:04.902 info: Remote account https://mastodon.social/users/jaspershailesh1 removed.                    
[tube.bitwaves.de:443] 2024-09-06 08:59:48.787 info: Moving video 0a55c308-3f4a-4064-90b8-6a3c3f4e7a87 to file system in job 1.               
[tube.bitwaves.de:443] 2024-09-06 08:59:48.901 info: Fetching HLS file hls/0a55c308-3f4a-4064-90b8-6a3c3f4e7a87/e14a8664-bdec-48a7-aa54-3e6387
d2af08-1080.m3u8 from object storage to /data/streaming-playlists/hls/0a55c308-3f4a-4064-90b8-6a3c3f4e7a87/e14a8664-bdec-48a7-aa54-3e6387d2af0
8-1080.m3u8.                                                                                                                                  
[tube.bitwaves.de:443] 2024-09-06 08:59:50.913 info: Initialized S3 client https://s3.hidrive.strato.com with region us-east-1.               
[tube.bitwaves.de:443] 2024-09-06 08:59:50.991 info: 2a02:26f7:ec60:6606:0:dad:1a33:c1f6 - - [06/Sep/2024:08:59:50 +0000] "GET / HTTP/1.0" 304
 - "-" "Mozilla/5.0 (iPad; CPU OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1"     
[tube.bitwaves.de:443] 2024-09-06 08:59:51.049 info: Fetching HLS file hls/0a55c308-3f4a-4064-90b8-6a3c3f4e7a87/e14a8664-bdec-48a7-aa54-3e6387
d2af08-1080-fragmented.mp4 from object storage to /data/streaming-playlists/hls/0a55c308-3f4a-4064-90b8-6a3c3f4e7a87/e14a8664-bdec-48a7-aa54-3
e6387d2af08-1080-fragmented.mp4.

normen commented 1 week ago

Obviously, after getting this issue twice I don't have it anymore now that I posted here. The only thing I changed was renaming my videos folder to web-videos - the internal names of the web_videos variables were already changed, just the actual folders in s3 and on the server were still named videos. Could that have been it?

normen commented 1 week ago

On some videos I get legit errors where the job state is "failed", the log looks like this:

ooks.js:8:24)\n    at async Worker.processJob (/app/node_modules/bullmq/dist/cjs/classes/worker.js:440:28)\n    at async Worker.retryIfFailed (/app/node_modules/bullmq/dist/cjs/classes/worker.js:629:24)",
    "message": "The \"path\" argument must be of type string. Received null",
    "code": "ERR_INVALID_ARG_TYPE"
  }
}
[tube.bitwaves.de:443] 2024-09-06 10:42:05.467 error: Cannot execute job 12 in queue move-to-file-system. {
  "payload": {
    "videoUUID": "5f5c6c2b-84c2-47c6-ae7e-7e2845b28a50",
    "isNewVideo": false,
    "previousVideoState": 11
  },
  "err": {
    "stack": "TypeError [ERR_INVALID_ARG_TYPE]: The \"path\" argument must be of type string. Received null\n    at new NodeError (node:internal/errors:405:5)\n    at validateString (node:internal/validators:162:11)\n    at join (node:path:1171:7)\n    at VideoPathManager.getFSOriginalVideoFilePath (file:///app/dist/core/lib/video-path-manager.js:42:16)\n    at moveVideoSourceFile (file:///app/dist/core/lib/job-queue/handlers/move-to-file-system.js:38:92)\n    at moveToJob (file:///app/dist/core/lib/job-queue/handlers/shared/move-video.js:21:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.processMoveToFileSystem [as move-to-file-system] (file:///app/dist/core/lib/job-queue/handlers/move-to-file-system.js:15:5)\n    at async Object.wrapPromiseFun (file:///app/dist/core/lib/plugins/hooks.js:8:24)\n    at async Worker.processJob (/app/node_modules/bullmq/dist/cjs/classes/worker.js:440:28)\n    at async Worker.retryIfFailed (/app/node_modules/bullmq/dist/cjs/classes/worker.js:629:24)",
    "message": "The \"path\" argument must be of type string. Received null",
    "code": "ERR_INVALID_ARG_TYPE"
  }
}

I don't know how to cancel this, I always get a warning about the failed download on that video page.

Chocobozzz commented 3 days ago

Hi,

Do you reproduce your issue with new videos?

normen commented 3 days ago

Yes, I got this from a video I just uploaded with the same peertube version:

[tube.bitwaves.de:443] 2024-09-13 08:15:39.507 error: Cannot move video https://tube.bitwaves.de/videos/watch/9813a2da-5a5a-468e-a1ab-ced22e2b325f storage. {
  "err": {
    "stack": "TypeError [ERR_INVALID_ARG_TYPE]: The \"path\" argument must be of type string. Received null\n    at new NodeError (node:internal/errors:405:5)\n    at validateString (node:internal/validators:162:11)\n    at join (node:path:1171:7)\n    at VideoPathManager.getFSOriginalVideoFilePath (file:///app/dist/core/lib/video-path-manager.js:42:16)\n    at moveVideoSourceFile (file:///app/dist/core/lib/job-queue/handlers/move-to-file-system.js:38:92)\n    at moveToJob (file:///app/dist/core/lib/job-queue/handlers/shared/move-video.js:21:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.processMoveToFileSystem [as move-to-file-system] (file:///app/dist/core/lib/job-queue/handlers/move-to-file-system.js:15:5)\n    at async Object.wrapPromiseFun (file:///app/dist/core/lib/plugins/hooks.js:8:24)\n    at async Worker.processJob (/app/node_modules/bullmq/dist/cjs/classes/worker.js:440:28)\n    at async Worker.retryIfFailed (/app/node_modules/bullmq/dist/cjs/classes/worker.js:629:24)",
    "message": "The \"path\" argument must be of type string. Received null",
    "code": "ERR_INVALID_ARG_TYPE"
  }
}
[tube.bitwaves.de:443] 2024-09-13 08:15:39.817 error: Cannot execute job 17 in queue move-to-file-system. {
  "payload": {
    "videoUUID": "9813a2da-5a5a-468e-a1ab-ced22e2b325f",
    "isNewVideo": false,
    "previousVideoState": 1
  },
  "err": {
    "stack": "TypeError [ERR_INVALID_ARG_TYPE]: The \"path\" argument must be of type string. Received null\n    at new NodeError (node:internal/errors:405:5)\n    at validateString (node:internal/validators:162:11)\n    at join (node:path:1171:7)\n    at VideoPathManager.getFSOriginalVideoFilePath (file:///app/dist/core/lib/video-path-manager.js:42:16)\n    at moveVideoSourceFile (file:///app/dist/core/lib/job-queue/handlers/move-to-file-system.js:38:92)\n    at moveToJob (file:///app/dist/core/lib/job-queue/handlers/shared/move-video.js:21:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.processMoveToFileSystem [as move-to-file-system] (file:///app/dist/core/lib/job-queue/handlers/move-to-file-system.js:15:5)\n    at async Object.wrapPromiseFun (file:///app/dist/core/lib/plugins/hooks.js:8:24)\n    at async Worker.processJob (/app/node_modules/bullmq/dist/cjs/classes/worker.js:440:28)\n    at async Worker.retryIfFailed (/app/node_modules/bullmq/dist/cjs/classes/worker.js:629:24)",
    "message": "The \"path\" argument must be of type string. Received null",
    "code": "ERR_INVALID_ARG_TYPE"
  }
}

However live stream recordings don't seem to have that issue, only video uploads. Automatic recordings of permanent live streams download fine. I do not have transcoding enabled for lives on the server.

normen commented 3 days ago

And by the way to get rid of the "moving to storage failed" message on the video, that remains after the above error, I have to transcode to web video once and then delete it again. After that the message disappears.

normen commented 3 days ago

Another by the way: It seems that even though I have web video disabled the server still creates web video files for imports. Later when I do a prune-storage a lot of web-video files are found that can be deleted.. Maybe thats the "path" it can't find?

Chocobozzz / PeerTube