Unbounded latency/memory in live streams

[ghost]

Describe the current behavior

Not sure which specific situations this bug occurs in; it might be only when the CPU is loaded. I did a live stream that lasted a total of 5.5 hours, at 1080p. During the stream someone measured the lag as around 30 minutes. By the end of the broadcast, PeerTube is still playing buffered video, and the nodejs process running the media server has 15 gigs of ram resident.

Steps to reproduce:

1. Live stream
2. Let it run for a while
3. Check lag & memory usage

Describe the expected behavior

I expect the latency to be bounded; if the encoder falls behind for whatever reason, keyframes should drop rather than the buffer growing unbounded.

Additional information

• PeerTube instance:

  • URL: diode.zone
  • version: 3.0.0
  • NodeJS version: 10.23.1
  • Ffmpeg version: git

• Browser name, version and platforms on which you could reproduce the bug:

• Link to browser console log if relevant:

• Link to server log if relevant (journalctl or /var/www/peertube/storage/logs/):

[ghost]

I noticed a similar lag issue in a continuous stream that I have running. This stream has a clock in the corner which should roughly match local time, and it's several hours off by now. I'm going to have to restart the server probably, because it's still using 16 gigs of RAM.

https://diode.zone/videos/watch/6573fd5e-da3d-476f-b002-3ff5b4a12d0e

[test2a]

Shameless plug but I set up a repo to collect stats and logs on lives so that the team can get a single place of lots of data. No one has yet actually started but maybe you guys can start?

https://github.com/test2a/peertubestresstest

[ghost]

Shameless plug but I set up a repo to collect stats and logs on lives so that the team can get a single place of lots of data. No one has yet actually started but maybe you guys can start?

https://github.com/test2a/peertubestresstest

I'll check it out. My server has webtorrent disabled (to save storage) for what it's worth, not sure if those tests are mandatory.

Update: It might be useful to think about the specific resources that are a bottleneck in common configurations. On my physical server at least, i have come nowhere close to using the available disk or network bandwidth, and there's also plenty of ram, but the performance issues often stem from excess work or improper prioritization of work on the cpu.

For example, short of an intentional DoS attack i'd have a hard time imagining downloads or static files ever being a significant limitation. But sometimes the API server bogs down, and transcoding is always a bottleneck. All transcoding processes right now are fixed at the same cpu priority, even though live transcodes should be higher than background jobs. And background jobs can't be cancelled cleanly.

[test2a]

@scanlime oh thats not an issue. you can specify your own tests, workflows and such. its just what came to my mind...

[Chocobozzz]

Should be fixed by https://github.com/Chocobozzz/PeerTube/commit/00b87c5791ecd645bb78cbb9872d60e1f957bdfa

If ffmpeg is too slow to handle transcode the stream, we abort the session.