Error polling Segment: SegmenterTimeout when RTMP content input with frame rates <=25

[chrishobcroft]

Summary

When streaming RTMP content into livepeer in broadcaster mode, the streams fail when the content is <= 25 frames per second.

The failure is reported as a SegmenterTimeout.

Streams of RTMP content with >25 FPS are processed successfully.

More Detail

When using version 0.4.0, started with ./livepeer -offchain -httpAddr 0.0.0.0:8935 -rtmpAddr 0.0.0.0:1935 -currentManifest

SegmenterTimeout errors when broadcasting from OBS at FPS of presets 24 NTSC, 25 PAL, 20, 10, and manually set 25, 24, 15, 1.

NO SegmenterTimeout errors when broadcasting from OBS at FPS of presets 60, 59.94, 50 PAL, 48, 30, 29.97 and manually set: 29, 28, 27, 26

Error Log

E0705 14:31:02.641940    2867 lpms.go:172] Error polling Segment: SegmenterTimeout, Retrying
E0705 14:31:03.142456    2867 lpms.go:172] Error polling Segment: SegmenterTimeout, Retrying
E0705 14:31:03.642891    2867 lpms.go:198] Error segmenting stream: SegmenterTimeout
E0705 14:31:03.643307    2867 basic_rtmp_videostream.go:109] Error reading packet from RTMP: read tcp 178.62.235.51:1935->2.4.142.235:55648: use of closed network connection

[j0sh]

Thanks for the report. This seems similar to an earlier issue from a while ago - https://github.com/livepeer/go-livepeer/issues/328 ?

In any case, @darkdarkdragon has some fixes coming up soon which should help with this type of issue. Hang tight!

[darkdarkdragon]

Can't reproduce that. @chrishobcroft can you check with latest build? https://build.livepeer.live/0.3.3-9b95737b/livepeer-darwin-amd64.tar.gz https://build.livepeer.live/0.3.3-9b95737b/livepeer-linux-amd64.tar.gz

[chrishobcroft]

@darkdarkdragon, I am also able to reproduce with those builds on a clean Ubuntu 18.04, when streaming in at 10fps from OBS. (Streaming in at 30fps from OBS works fine - see below).

Here is the transcript:

chrishobcroft@livepeer-broadcaster:~/livepeer-linux-amd64$ ./livepeer -version
Livepeer Node Version: undefined
Golang runtime version: gc go1.12
Architecture: amd64
Operating system: linux
chrishobcroft@livepeer-broadcaster:~/livepeer-linux-amd64$ ./livepeer -broadcaster -rtmpAddr 0.0.0.0:1935 -httpAddr 0.0.0.0:8935 -currentManifest
I0706 09:53:51.316046    7902 livepeer.go:183] ***Livepeer is running on the offchain*** network
I0706 09:53:51.317393    7902 livepeer.go:272] ***Livepeer is in off-chain mode***
E0706 09:53:51.317564    7902 livepeer.go:490] No orchestrator specified; transcoding will not happen
I0706 09:53:51.317754    7902 livepeer.go:546] Current ManifestID will be available over 0.0.0.0:8935
I0706 09:53:51.317953    7902 livepeer.go:585] ***Livepeer Running in Broadcaster Mode***
I0706 09:53:51.318142    7902 livepeer.go:586] Video Ingest Endpoint - rtmp://0.0.0.0:1935
I0706 09:53:51.318384    7902 webserver.go:65] CLI server listening on 127.0.0.1:7935
I0706 09:54:02.142153    7902 broadcast.go:176] No orchestrators specified; not transcoding
I0706 09:54:02.142654    7902 mediaserver.go:304] 

Video Created With ManifestID: f

[h264 @ 0x7fbee4007880] co located POCs unavailable
E0706 09:54:22.282477    7902 lpms.go:174] Error polling Segment: SegmenterTimeout, Retrying
E0706 09:54:22.782980    7902 lpms.go:174] Error polling Segment: SegmenterTimeout, Retrying
E0706 09:54:23.283745    7902 lpms.go:200] Error segmenting stream: SegmenterTimeout
E0706 09:54:23.284090    7902 basic_rtmp_videostream.go:135] Error reading packet from RTMP: read tcp 178.62.235.51:1935->2.4.142.235:48198: use of closed network connection
I0706 09:54:23.284286    7902 mediaserver.go:327] Ended stream with id=f

And here is the screenshot showing OBS Disconnecting.

...and when streaming in at 30fps, everything seems to work fine.

Here is the transcript:

chrishobcroft@livepeer-broadcaster:~/livepeer-linux-amd64$ ./livepeer -version
Livepeer Node Version: undefined
Golang runtime version: gc go1.12
Architecture: amd64
Operating system: linux
chrishobcroft@livepeer-broadcaster:~/livepeer-linux-amd64$ ./livepeer -broadcaster -rtmpAddr 0.0.0.0:1935 -httpAddr 0.0.0.0:8935 -currentManifest
I0706 09:57:22.563944    7919 livepeer.go:183] ***Livepeer is running on the offchain*** network
I0706 09:57:22.565480    7919 livepeer.go:272] ***Livepeer is in off-chain mode***
E0706 09:57:22.565693    7919 livepeer.go:490] No orchestrator specified; transcoding will not happen
I0706 09:57:22.565889    7919 livepeer.go:546] Current ManifestID will be available over 0.0.0.0:8935
I0706 09:57:22.566074    7919 livepeer.go:585] ***Livepeer Running in Broadcaster Mode***
I0706 09:57:22.566245    7919 livepeer.go:586] Video Ingest Endpoint - rtmp://0.0.0.0:1935
I0706 09:57:22.566467    7919 webserver.go:65] CLI server listening on 127.0.0.1:7935
I0706 09:57:29.494324    7919 broadcast.go:176] No orchestrators specified; not transcoding
I0706 09:57:29.494592    7919 mediaserver.go:304] 

Video Created With ManifestID: f

I0706 09:57:46.726886    7919 broadcast.go:287] No sessions available for segment nonce=4804146931952328171 seqNo=0
I0706 09:57:55.645559    7919 broadcast.go:287] No sessions available for segment nonce=4804146931952328171 seqNo=1
I0706 09:58:04.261687    7919 broadcast.go:287] No sessions available for segment nonce=4804146931952328171 seqNo=2
I0706 09:58:13.077673    7919 broadcast.go:287] No sessions available for segment nonce=4804146931952328171 seqNo=3
I0706 09:58:19.290019    7919 broadcast.go:287] No sessions available for segment nonce=4804146931952328171 seqNo=4
I0706 09:58:28.108688    7919 broadcast.go:287] No sessions available for segment nonce=4804146931952328171 seqNo=5
I0706 09:58:29.509985    7919 mediaserver.go:327] Ended stream with id=f

And here is the proof that ffplay works for playing back the stream locally.

[chrishobcroft]

Update on this - when setting OBS to stream content with a keyframe interval of 1s, I was able to reduce the frame rate to 4fps successfully.

3fps and below still resulted in errors, and I was not able to reduce the keyframe interval any lower than this.

So, it appears to remain the case that Livepeer cannot handle streams with low fps, which might rule it out from e.g. some CCTV video processing, or other video which do not have higher frame rates.

Interested to know from @f1l1b0x what use cases there might be for low fps video transcoding?

[chrishobcroft]

I am seeing this now using Manycam on Android.m to stream into 1935