search

Chocobozzz / PeerTube

ActivityPub-federated video streaming platform using P2P directly in your web browser
https://joinpeertube.org/
GNU Affero General Public License v3.0
13.09k stars 1.51k forks source link

Separate storage path for live streaming #5981

Closed jb-alvarado closed 1 year ago

jb-alvarado commented 1 year ago

Describe the problem to be solved

When I test the live streaming function, one time peertube crashes and restart and I get that errors:

Sep 20 17:18:12 tube1 peertube[61053]: [tube.example.org:443] 2023-09-20 17:18:12.988 info: 111.222.0.0 - - [20/Sep/2023:15:18:12 +0000] "POST /api/v1/server/logs/client HTTP/1.0" 204 - "https://tube.example.org/w/juFLUtM1aCx54qAhbkB7up" "Firefox"
Sep 20 17:18:14 tube1 peertube[61053]: [tube.example.org:443] 2023-09-20 17:18:14.049 error: Client log: Segment https://tube.example.org/static/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/master.m3u8+V3+118 error. {
Sep 20 17:18:14 tube1 peertube[61053]:   "username": "daswort",
Sep 20 17:18:14 tube1 peertube[61053]:   "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0",
Sep 20 17:18:14 tube1 peertube[61053]:   "meta": "{\"isTrusted\":true}",
Sep 20 17:18:14 tube1 peertube[61053]:   "url": "https://tube.example.org/w/juFLUtM1aCx54qAhbkB7up"
Sep 20 17:18:14 tube1 peertube[61053]: }
Sep 20 17:18:14 tube1 peertube[61053]: [tube.example.org:443] 2023-09-20 17:18:14.054 info: 111.222.0.0 - - [20/Sep/2023:15:18:14 +0000] "POST /api/v1/server/logs/client HTTP/1.0" 204 - "https://tube.example.org/w/juFLUtM1aCx54qAhbkB7up" "Firefox"
Sep 20 17:18:15 tube1 peertube[61053]: [tube.example.org:443] 2023-09-20 17:18:15.661 error: uncaughtException: UNKNOWN: unknown error, watch '/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json'
Sep 20 17:18:15 tube1 peertube[61053]: Error: UNKNOWN: unknown error, watch '/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json'
Sep 20 17:18:15 tube1 peertube[61053]:     at FSWatcher.<computed> (node:internal/fs/watchers:244:19)
Sep 20 17:18:15 tube1 peertube[61053]:     at Object.watch (node:fs:2315:34)
Sep 20 17:18:15 tube1 peertube[61053]:     at createFsWatchInstance (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:119:15)
Sep 20 17:18:15 tube1 peertube[61053]:     at setFsWatchListener (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:166:15)
Sep 20 17:18:15 tube1 peertube[61053]:     at NodeFsHandler._watchWithNodeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:331:14)
Sep 20 17:18:15 tube1 peertube[61053]:     at NodeFsHandler._handleFile (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:395:23)
Sep 20 17:18:15 tube1 peertube[61053]:     at NodeFsHandler._addToNodeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:637:21) {
Sep 20 17:18:15 tube1 peertube[61053]:   "error": {
Sep 20 17:18:15 tube1 peertube[61053]:     "stack": "Error: UNKNOWN: unknown error, watch '/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json'\n    at FSWatcher.<computed> (node:internal/fs/watchers:244:19)\n    at Object.wat
ch (node:fs:2315:34)\n    at createFsWatchInstance (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:119:15)\n    at setFsWatchListener (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:166:15)\n    at Node
FsHandler._watchWithNodeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:331:14)\n    at NodeFsHandler._handleFile (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:395:23)\n    at NodeFsHandler._addToN
odeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:637:21)",
Sep 20 17:18:15 tube1 peertube[61053]:     "message": "UNKNOWN: unknown error, watch '/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json'",
Sep 20 17:18:15 tube1 peertube[61053]:     "errno": -116,
Sep 20 17:18:15 tube1 peertube[61053]:     "syscall": "watch",
Sep 20 17:18:15 tube1 peertube[61053]:     "code": "UNKNOWN",
Sep 20 17:18:15 tube1 peertube[61053]:     "path": "/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json",
Sep 20 17:18:15 tube1 peertube[61053]:     "filename": "/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json"
Sep 20 17:18:15 tube1 peertube[61053]:   },
Sep 20 17:18:15 tube1 peertube[61053]:   "stack": "Error: UNKNOWN: unknown error, watch '/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json'\n    at FSWatcher.<computed> (node:internal/fs/watchers:244:19)\n    at Object.watch (node:fs:2315:34)\n    at createFsWatchInstance (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:119:15)\n    at setFsWatchListener (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:166:15)\n    at NodeFsHandler._watchWithNodeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:331:14)\n    at NodeFsHandler._handleFile (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:395:23)\n    at NodeFsHandler._addToNodeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:637:21)",
Sep 20 17:18:15 tube1 peertube[61053]:   "exception": true,
Sep 20 17:18:15 tube1 peertube[61053]:   "date": "Wed Sep 20 2023 17:18:15 GMT+0200 (Mitteleuropäische Sommerzeit)",
Sep 20 17:18:15 tube1 peertube[61053]:   "process": {
Sep 20 17:18:15 tube1 peertube[61053]:     "pid": 61053,
Sep 20 17:18:15 tube1 peertube[61053]:     "uid": 1001,
Sep 20 17:18:15 tube1 peertube[61053]:     "gid": 1001,
Sep 20 17:18:15 tube1 peertube[61053]:     "cwd": "/var/www/peertube/versions/peertube-v5.2.1",
Sep 20 17:18:15 tube1 peertube[61053]:     "execPath": "/usr/bin/node",
Sep 20 17:18:15 tube1 peertube[61053]:     "version": "v18.13.0",
Sep 20 17:18:15 tube1 peertube[61053]:     "argv": [
Sep 20 17:18:15 tube1 peertube[61053]:       "/usr/bin/node",
Sep 20 17:18:15 tube1 peertube[61053]:       "/var/www/peertube/versions/peertube-v5.2.1/dist/server"
Sep 20 17:18:15 tube1 peertube[61053]:     ],
Sep 20 17:18:15 tube1 peertube[61053]:     "memoryUsage": {
Sep 20 17:18:15 tube1 peertube[61053]:       "rss": 534278144,
Sep 20 17:18:15 tube1 peertube[61053]:       "heapTotal": 165752832,
Sep 20 17:18:15 tube1 peertube[61053]:       "heapUsed": 136567952,
Sep 20 17:18:15 tube1 peertube[61053]:       "external": 25162359,
Sep 20 17:18:15 tube1 peertube[61053]:       "arrayBuffers": 22452668
Sep 20 17:18:15 tube1 peertube[61053]:     }
Sep 20 17:18:15 tube1 peertube[61053]:   },
Sep 20 17:18:15 tube1 peertube[61053]:   "os": {
Sep 20 17:18:15 tube1 peertube[61053]:     "loadavg": [
Sep 20 17:18:15 tube1 peertube[61053]:       3.33,
Sep 20 17:18:15 tube1 peertube[61053]:       3.49,
Sep 20 17:18:15 tube1 peertube[61053]:       2.09
Sep 20 17:18:15 tube1 peertube[61053]:     ],
Sep 20 17:18:15 tube1 peertube[61053]:     "uptime": 450109.2
Sep 20 17:18:15 tube1 peertube[61053]:   },
Sep 20 17:18:15 tube1 peertube[61053]:   "trace": [
Sep 20 17:18:15 tube1 peertube[61053]:     {
Sep 20 17:18:15 tube1 peertube[61053]:       "column": 19,
Sep 20 17:18:15 tube1 peertube[61053]:       "file": "node:internal/fs/watchers",
Sep 20 17:18:15 tube1 peertube[61053]:       "function": "FSWatcher.<computed>",
Sep 20 17:18:15 tube1 peertube[61053]:       "line": 244,
Sep 20 17:18:15 tube1 peertube[61053]:       "method": "<computed>",
Sep 20 17:18:15 tube1 peertube[61053]:       "native": false
Sep 20 17:18:15 tube1 peertube[61053]:     },
Sep 20 17:18:15 tube1 peertube[61053]:     {
Sep 20 17:18:15 tube1 peertube[61053]:       "column": 34,
Sep 20 17:18:15 tube1 peertube[61053]:       "file": "node:fs",
Sep 20 17:18:15 tube1 peertube[61053]:       "function": "Object.watch",
Sep 20 17:18:15 tube1 peertube[61053]:       "line": 2315,
Sep 20 17:18:15 tube1 peertube[61053]:       "method": "watch",
Sep 20 17:18:15 tube1 peertube[61053]:       "native": false
Sep 20 17:18:15 tube1 peertube[61053]:     },
Sep 20 17:18:15 tube1 peertube[61053]:     {
Sep 20 17:18:15 tube1 peertube[61053]:       "column": 15,
Sep 20 17:18:15 tube1 peertube[61053]:       "file": "/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js",
Sep 20 17:18:15 tube1 peertube[61053]:       "function": "createFsWatchInstance",
Sep 20 17:18:15 tube1 peertube[61053]:       "line": 119,
Sep 20 17:18:15 tube1 peertube[61053]:       "method": null,
Sep 20 17:18:15 tube1 peertube[61053]:       "native": false
Sep 20 17:18:15 tube1 peertube[61053]:     },
Sep 20 17:18:15 tube1 peertube[61053]:     {
Sep 20 17:18:15 tube1 peertube[61053]:       "column": 15,
Sep 20 17:18:15 tube1 peertube[61053]:       "file": "/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js",
Sep 20 17:18:15 tube1 peertube[61053]:       "function": "setFsWatchListener",
Sep 20 17:18:15 tube1 peertube[61053]:       "line": 166,
Sep 20 17:18:15 tube1 peertube[61053]:       "method": null,
Sep 20 17:18:15 tube1 peertube[61053]:       "native": false
Sep 20 17:18:15 tube1 peertube[61053]:     },
Sep 20 17:18:15 tube1 peertube[61053]:     {
Sep 20 17:18:15 tube1 peertube[61053]:       "column": 14,
Sep 20 17:18:15 tube1 peertube[61053]:       "file": "/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js",
Sep 20 17:18:15 tube1 peertube[61053]:       "function": "NodeFsHandler._watchWithNodeFs",
Sep 20 17:18:15 tube1 peertube[61053]:       "line": 331,
Sep 20 17:18:15 tube1 peertube[61053]:       "method": "_watchWithNodeFs",
Sep 20 17:18:15 tube1 peertube[61053]:       "native": false
Sep 20 17:18:15 tube1 peertube[61053]:     },
Sep 20 17:18:15 tube1 peertube[61053]:     {
Sep 20 17:18:15 tube1 peertube[61053]:       "column": 23,
Sep 20 17:18:15 tube1 peertube[61053]:       "file": "/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js",
Sep 20 17:18:15 tube1 peertube[61053]:       "function": "NodeFsHandler._handleFile",
Sep 20 17:18:15 tube1 peertube[61053]:       "line": 395,
Sep 20 17:18:15 tube1 peertube[61053]:       "method": "_handleFile",
Sep 20 17:18:15 tube1 peertube[61053]:       "native": false
Sep 20 17:18:15 tube1 peertube[61053]:     },
Sep 20 17:18:15 tube1 peertube[61053]:     {
Sep 20 17:18:15 tube1 peertube[61053]:       "column": 21,
Sep 20 17:18:15 tube1 peertube[61053]:       "file": "/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js",
Sep 20 17:18:15 tube1 peertube[61053]:       "function": "NodeFsHandler._addToNodeFs",
Sep 20 17:18:15 tube1 peertube[61053]:       "line": 637,
Sep 20 17:18:15 tube1 peertube[61053]:       "method": "_addToNodeFs",
Sep 20 17:18:15 tube1 peertube[61053]:       "native": false
Sep 20 17:18:15 tube1 peertube[61053]:     }
Sep 20 17:18:15 tube1 peertube[61053]:   ]
Sep 20 17:18:15 tube1 peertube[61053]: }
Sep 20 17:18:15 tube1 peertube[61053]: [tube.example.org:443 uploadx] 2023-09-20 17:18:15.663 error: uncaughtException: UNKNOWN: unknown error, watch '/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json'
Sep 20 17:18:15 tube1 peertube[61053]: Error: UNKNOWN: unknown error, watch '/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json'
Sep 20 17:18:15 tube1 peertube[61053]:     at FSWatcher.<computed> (node:internal/fs/watchers:244:19)
Sep 20 17:18:15 tube1 peertube[61053]:     at Object.watch (node:fs:2315:34)
Sep 20 17:18:15 tube1 peertube[61053]:     at createFsWatchInstance (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:119:15)
Sep 20 17:18:15 tube1 peertube[61053]:     at setFsWatchListener (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:166:15)
Sep 20 17:18:15 tube1 peertube[61053]:     at NodeFsHandler._watchWithNodeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:331:14)
Sep 20 17:18:15 tube1 peertube[61053]:     at NodeFsHandler._handleFile (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:395:23)
Sep 20 17:18:15 tube1 peertube[61053]:     at NodeFsHandler._addToNodeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:637:21) {
Sep 20 17:18:15 tube1 peertube[61053]:   "error": {
Sep 20 17:18:15 tube1 peertube[61053]:     "stack": "Error: UNKNOWN: unknown error, watch '/var/www/peertube/storage/streaming-playlists/hls/95c5d54a-e95b-4f8b-ad06-85b09edbd02f/segments-sha256.json'\n    at FSWatcher.<computed> (node:internal/fs/watchers:244:19)\n    at Object.watch (node:fs:2315:34)\n    at createFsWatchInstance (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:119:15)\n    at setFsWatchListener (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:166:15)\n    at NodeFsHandler._watchWithNodeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:331:14)\n    at NodeFsHandler._handleFile (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:395:23)\n    at NodeFsHandler._addToNodeFs (/var/www/peertube/versions/peertube-v5.2.1/node_modules/chokidar/lib/nodefs-handler.js:637:21)",

As I understand from other issues here, that can happen when using NFS (slow) storage. I using NFS, because I have no other option, but the NFS Server runs on a dedicated host and Peertube runs as a VM on that same host. In normal situations that setup runs very smooth, without issues.

Describe the solution you would like

For Object Storage there is an option, that after transcoding the files are moved to the new storage. How about a similar function for live streaming. That we can have a separate storage path for live streaming, like: /var/www/peertube/storage/live/, and when the live stream stops and the stream got prepared for VOD it moves to the default location /var/www/peertube/storage/streaming-playlists/.

That would make it possible to use a smaller and faster storage for live straming and a bigger and slower storage for VOD. Even a Ram Disk could be possible for that use case.

Chocobozzz commented 1 year ago

Thanks for reporting this issue, but this is a duplicate of https://github.com/Chocobozzz/PeerTube/issues/5843