Live stream freezes after a while

fabiopicchi commented 3 months ago

Describe the current behavior

Streams seem to be working just fine for a short while. Once they get to 10 or so minutes they freeze, at least for me. I am not sure if this is the underlying cause, but after roughly 10 attempts, I see a storm of errors like this in the logs:

{"level":"error","message":"Cannot copy segment /var/www/peertube/storage/streaming-playlists/hls/4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8/3-000240.ts to replay directory.","label":"play.cotv.org.br:443","err":{"stack":"Error: ENOENT: no such file or directory, open '/var/www/peertube/storage/streaming-playlists/hls/4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8/3-000240.ts'\n    at async open (node:internal/fs/promises:641:25)\n    at async readFile (node:internal/fs/promises:1254:14)\n    at async MuxingSession.addSegmentToReplay (file:///var/www/peertube/versions/peertube-v6.1.0/dist/core/lib/live/shared/muxing-session.js:283:26)\n    at async MuxingSession.processSegment (file:///var/www/peertube/versions/peertube-v6.1.0/dist/core/lib/live/shared/muxing-session.js:207:13)","message":"ENOENT: no such file or directory, open '/var/www/peertube/storage/streaming-playlists/hls/4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8/3-000240.ts'","errno":-2,"code":"ENOENT","syscall":"open","path":"/var/www/peertube/storage/streaming-playlists/hls/4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8/3-000240.ts"},"tags":["live","UGUZ1MQV","4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8"],"timestamp":"2024-06-29T16:34:19.936Z"}
{"level":"error","message":"Cannot store TS segment /var/www/peertube/storage/streaming-playlists/hls/4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8/3-000240.ts in object storage","label":"play.cotv.org.br:443","err":{"stack":"Error: ENOENT: no such file or directory, open '/var/www/peertube/storage/streaming-playlists/hls/4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8/3-000240.ts'","message":"ENOENT: no such file or directory, open '/var/www/peertube/storage/streaming-playlists/hls/4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8/3-000240.ts'","errno":-2,"code":"ENOENT","syscall":"open","path":"/var/www/peertube/storage/streaming-playlists/hls/4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8/3-000240.ts"},"tags":["live","UGUZ1MQV","4f8d1b8f-3cfe-4795-b1b9-9b815fb1f4d8"],"timestamp":"2024-06-29T16:34:19.937Z"}

My suspicion is that the ts file cleanup is mistakenly deleting files it shouldn't.

Steps to reproduce

Start the stream
Stream for a while
Notice the errors in the logs and the stream freezes

Describe the expected behavior

The stream should work uninterruptedly.

Additional information

PeerTube instance:
- URL: play.cotv.org.br
- Version: 6.1.0
- NodeJS version: v22.3.0
- Ffmpeg version: ffmpeg version 6.1.1-3ubuntu5 Copyright (c) 2000-2023 the FFmpeg developers built with gcc 13 (Ubuntu 13.2.0-23ubuntu3) configuration: --prefix=/usr --extra-version=3ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --disable-omx --enable-gnutls --enable-libaom --enable-libass --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libharfbuzz --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-openal --enable-opencl --enable-opengl --disable-sndio --enable-libvpl --disable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-ladspa --enable-libbluray --enable-libjack --enable-libpulse --enable-librabbitmq --enable-librist --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libx264 --enable-libzmq --enable-libzvbi --enable-lv2 --enable-sdl2 --enable-libplacebo --enable-librav1e --enable-pocketsphinx --enable-librsvg --enable-libjxl --enable-shared libavutil 58. 29.100 / 58. 29.100 libavcodec 60. 31.102 / 60. 31.102 libavformat 60. 16.100 / 60. 16.100 libavdevice 60. 3.100 / 60. 3.100 libavfilter 9. 12.100 / 9. 12.100 libswscale 7. 5.100 / 7. 5.100 libswresample 4. 12.100 / 4. 12.100 libpostproc 57. 3.100 / 57. 3.100
Browser name, version and platforms on which you could reproduce the bug:
Link to browser console log if relevant:
Link to server log if relevant (journalctl or /var/www/peertube/storage/logs/):

fabiopicchi commented 3 months ago

Disabling external storage seems to have circumvented the issue, but we need object storage in production so it is just a temporary solution.

On a side note: I am using BackBlaze.

Chocobozzz commented 3 months ago

Hi,

Can you enable debug logs in your configuration, restart peertube and reproduce? I would need more context around the errors :)

fabiopicchi commented 3 months ago

Sure, I'll just ask for a bit of time as I will have to do this setup locally since production is working at the moment. I toggled this flag in the config file:

    # PeerTube makes many small requests to the object storage provider to upload/delete/update live chunks
    # which can be a problem depending on your object storage provider
    # You can also choose to disable this feature to reduce live streams latency
    # Live stream replays are not affected by this setting, so they are uploaded in object storage as regular VOD videos
    store_live_streams: false

Chocobozzz commented 2 months ago

Hi @fabiopicchi

Do you have the same error in peertube 6.2? object_storage.max_request_attempts introduced in this version may help since you're using blackbaze

fabiopicchi commented 2 months ago

Thanks for letting me know @Chocobozzz ! I tried using this flag inadvertently in 6.1 😅 I'll let you know when I update it. Sorry for not coming back with debug logs. I got drowned in the mega import we had to work on. The setting to allow us to only upload after the live has ended has been serving us well, though.

Chocobozzz commented 2 weeks ago

Closing due to inactivity, but feel free to comment!

kontrollanten commented 2 weeks ago

I've a similar issue in 6.2.1:

{
    "level": "error",
    "message": "Cannot copy segment /peertube/storage/streaming-playlists/hls/7184e230-f7f9-4338-9417-df878db552fc/1-000384.ts to replay directory.",
    "err": {
        "stack": "Error: ENOENT: no such file or directory, open '/peertube/storage/streaming-playlists/hls/7184e230-f7f9-4338-9417-df878db552fc/1-000384.ts'",
        "message": "ENOENT: no such file or directory, open '/peertube/storage/streaming-playlists/hls/7184e230-f7f9-4338-9417-df878db552fc/1-000384.ts'",
        "errno": -2,
        "code": "ENOENT",
        "syscall": "open",
        "path": "/peertube/storage/streaming-playlists/hls/7184e230-f7f9-4338-9417-df878db552fc/1-000384.ts"
    },
    "tags": [
        "live",
        "BPDSABRK",
        "7184e230-f7f9-4338-9417-df878db552fc"
    ],
    "timestamp": "2024-09-12T08:24:57.080Z"
}
{
    "tags": [
        "client"
    ],
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0",
    "stackTrace": "Unknown segment name 2-000385.ts in segment validator\nError: Unknown segment name 2-000385.ts in segment validator\n    at ml.<anonymous> (/client/sv-SE/chunk-ONRWSPRY.js:110:24365)\n    at Generator.next (<anonymous>)\n    at o (/client/sv-SE/chunk-ODKBFOSP.js:1:1257)\n    at o.invoke (/client/sv-SE/polyfills-DKLCR6XS.js:1:119357)\n    at D.run (/client/sv-SE/polyfills-DKLCR6XS.js:1:114647)\n    at /client/sv-SE/polyfills-DKLCR6XS.js:2:560\n    at o.invokeTask (/client/sv-SE/polyfills-DKLCR6XS.js:1:119985)\n    at D.runTask (/client/sv-SE/polyfills-DKLCR6XS.js:1:115264)\n    at F (/client/sv-SE/polyfills-DKLCR6XS.js:1:122044)",
    "level": "error",
    "message": "Client log: Segment /static/streaming-playlists/hls/7184e230-f7f9-4338-9417-df878db552fc/master.m3u8+V2+385 error.",
    "timestamp": "2024-09-12T08:24:57.086Z"
}

We've object_storage.streaming_playlists.store_live_streams set to false.

@Chocobozzz Any particular from the logs you're interested in?

Chocobozzz commented 2 weeks ago

@Chocobozzz Any particular from the logs you're interested in?

Yes, debug logs before and after these lines would help me

kontrollanten commented 2 weeks ago

I emailed you the logs.

Chocobozzz / PeerTube