Closed giulianoc closed 4 years ago
This usually happens when the receiver application doesn't read the packets fast enough.
But your case has something unusual in itself and it would be really interesting for us if you can reproduce it. It happens once per a time with very unusual and extraordinary conditions, may sometimes be connected with unfortunate thread scheduling and unusual data rates.
What I can see in these logs is that there are 1982 packets that the application should have read already a long time ago, and moreover, play at least most of them (the receiver buffer is used partially as latency buffer, but only some initial part of it). What I can see unusual though is that the distance between ACK sequence and packet sequence (see "offset") is larger than the buffer size (8192). It raises a suspicion that ACKs may be done very late - although it might be also a result of too long staying packet loss.
I would keenly see how the TSBPD and ACK times control happen (by a debug log) when this problem occurs.
Hi, in my previous ffmpeg command, the ts generated files were saved into a file system mounted through nfs. I thought that may be the nfs file system could create some problem in case of high throughput. For this reason I'm currently running the same command saving the ts files into a local file system. It seems I'm not having any problem but I need to wait a couple of days to be sure this change solved the issue. I'l let you know, best regards Giuliano
Hi @giulianoc. Please update the status of this issue.
I am also receiving this problem. I'm trying to run a push test to wowza. But testing locally sending mpegts with gstreamer I get the same problems.
I've had some serious issues trying to find the right commands to push mpegts or h264 to mpegts through SRT using gstreamer. This is what I have some up with so far which isn't anywhere to be found. I can't seem to get SRT working yet. SRT in Gstreamer is only found in source builds, the same might be for ffmpeg.
gst-launch-1.0 filesrc location=sintel_lang.ts ! tsparse set-timestamps=1 smoothing-latency=40000000 ! chopmydata step-size=188 min-size=188 max-size=1316 ! srtsink uri=srt://127.0.0.1:8081
23:30:53.384001*E: SRT.d: SND-DROPPED 24 packets - lost delaying for 1024ms
23:30:53.393016*E: SRT.d: SND-DROPPED 23 packets - lost delaying for 1023ms
23:30:53.402105*E: SRT.d: SND-DROPPED 23 packets - lost delaying for 1023ms
23:30:53.411108*E: SRT.d: SND-DROPPED 28 packets - lost delaying for 1023ms
On the listener
gst-play-1.0 srt://:8081
23:30:53.421856*E: SRT.c: %811065518:No room to store incoming packet: offset=31623 avail=6340 ack.seq=601059341 pkt.seq=601090964 rcv-remain=1851
23:30:53.422561*E: SRT.c: %811065518:No room to store incoming packet: offset=31624 avail=6340 ack.seq=601059341 pkt.seq=601090965 rcv-remain=1851
@danrossi Default receiver buffer size is approximately 96 Mbits. Please check this recommendation to see if the size is enough for your set up.
The receiver buffer must have enough space for (latency + RTT) * bitrate.
Hi. Some interesting tests. I managed to get ffmpeg compiled with srt on debian. ffmpeg as a caller to gstreamer on the same machine works fine. I get no such error.
ffmpeg -fflags +genpts -re -i sintel_lang.ts -acodec copy -vcodec copy -strict -2 -y -f mpegts 'srt://192.168.4.43:8081?pkt_size=1316&mode=caller'
ffmpeg -fflags +genpts -re -i sintel_lang_2000k.mp4 -acodec copy -vcodec copy -strict -2 -y -f mpegts 'srt://127.0.0.1:8081?pkt_size=1316&mode=caller'
Simple receiver command
gst-play-1.0 srt://:8081
Only when I use gstreamer as a caller do I get problems right away
gst-launch-1.0 filesrc location=sintel_lang.ts ! tsparse set-timestamps=1 smoothing-latency=40000000 ! chopmydata step-size=188 min-size=188 max-size=1316 ! srtsink uri=srt://127.0.0.1:8081
I have a similar problem. I am trying to do:
1 Haivision Media Gateway sending SRT to Amazon ubuntu server 2 ubuntu transcoding SRT to RTMP using ffmpeg ie:
ffmpeg -re -i "srt://0.0.0.0:9999?pkt_size=1316&mode=listener" -c:a aac -c:v h264 -g 48 -f flv "rtmp://localhost:1935/show/live"
(I need the RTMP for nginx which can use it to create a secure HLS stream)
Running at 4Mbit/s
root@ip-172-31-39-136:~/bin# ./ffmpeg -re -i "srt://0.0.0.0:9999?pkt_size=1316&mode=listener" -c:a aac -c:v h264 -g 48 -f flv "rtmp://localhost:1935/show/live"
ffmpeg version N-94396-g47b6ca0 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.11) 20160609
configuration: --prefix=/root/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --extra-libs='-lpthread -lm' --bindir=/root/bin --enable-gpl --enable-libaom --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libsrt --enable-nonfree
libavutil 56. 32.100 / 56. 32.100
libavcodec 58. 55.100 / 58. 55.100
libavformat 58. 30.100 / 58. 30.100
libavdevice 58. 9.100 / 58. 9.100
libavfilter 7. 58.100 / 7. 58.100
libswscale 5. 6.100 / 5. 6.100
libswresample 3. 6.100 / 3. 6.100
libpostproc 55. 6.100 / 55. 6.100
[h264 @ 0x2b0d600] sps_id 0 out of range
[h264 @ 0x2b0d600] SPS unavailable in decode_picture_timing
[h264 @ 0x2b0d600] non-existing PPS 0 referenced
[h264 @ 0x2b0d600] sps_id 0 out of range
[h264 @ 0x2b0d600] SPS unavailable in decode_picture_timing
[h264 @ 0x2b0d600] non-existing PPS 0 referenced
[h264 @ 0x2b0d600] decode_slice_header error
[h264 @ 0x2b0d600] no frame!
[h264 @ 0x2b0d600] sps_id 0 out of range
[h264 @ 0x2b0d600] SPS unavailable in decode_picture_timing
[h264 @ 0x2b0d600] non-existing PPS 0 referenced
[h264 @ 0x2b0d600] sps_id 0 out of range
[h264 @ 0x2b0d600] SPS unavailable in decode_picture_timing
[h264 @ 0x2b0d600] non-existing PPS 0 referenced
[h264 @ 0x2b0d600] decode_slice_header error
[h264 @ 0x2b0d600] no frame!
..... lots of these messages
[h264 @ 0x2b0d600] sps_id 0 out of range
[h264 @ 0x2b0d600] SPS unavailable in decode_picture_timing
[h264 @ 0x2b0d600] non-existing PPS 0 referenced
[h264 @ 0x2b0d600] sps_id 0 out of range
[h264 @ 0x2b0d600] SPS unavailable in decode_picture_timing
[h264 @ 0x2b0d600] non-existing PPS 0 referenced
[h264 @ 0x2b0d600] decode_slice_header error
[h264 @ 0x2b0d600] no frame!
Input #0, mpegts, from 'srt://0.0.0.0:9999?pkt_size=1316&mode=listener':
Duration: N/A, start: 8.319178, bitrate: N/A
Program 1
Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(top first), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
Stream #0:1[0x110](eng): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:1 -> #0:1 (mp2 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264 @ 0x2bf0e00] co located POCs unavailable
[h264 @ 0x2b429c0] co located POCs unavailable=-577014:32:22.77 bitrate= -0.0kbits/s speed=N/A
[libx264 @ 0x2b01e40] using SAR=1/1 0kB time=-577014:32:22.77 bitrate= -0.0kbits/s speed=N/A
[libx264 @ 0x2b01e40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x2b01e40] profile High, level 4.0
[libx264 @ 0x2b01e40] 264 - core 148 r2643 5c65704 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=48 keyint_min=4 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, flv, to 'rtmp://localhost:1935/show/live':
Metadata:
encoder : Lavf58.30.100
Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 25 fps, 1k tbn, 25 tbc
Metadata:
encoder : Lavc58.55.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
Stream #0:1(eng): Audio: aac (LC) ([10][0][0][0] / 0x000A), 48000 Hz, stereo, fltp, 128 kb/s
Metadata:
encoder : Lavc58.55.100 aac
.. Runs OK for about 20 seconds ..
Then:
10:49:20.015462/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=76 avail=62 ack.seq=1270402425 pkt.seq=1270402501 rcv-remain=8129
10:49:20.015675/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=77 avail=62 ack.seq=1270402425 pkt.seq=1270402502 rcv-remain=8129
10:49:20.053948/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=76 avail=74 ack.seq=1270402425 pkt.seq=1270402501 rcv-remain=8117
10:49:20.054367/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=77 avail=74 ack.seq=1270402425 pkt.seq=1270402502 rcv-remain=8117
10:49:20.064025/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=92 avail=74 ack.seq=1270402425 pkt.seq=1270402517 rcv-remain=8117
10:49:20.064091/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=93 avail=74 ack.seq=1270402425 pkt.seq=1270402518 rcv-remain=8117
10:49:20.087756/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=107 avail=74 ack.seq=1270402425 pkt.seq=1270402532 rcv-remain=8117
10:49:20.088917/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=108 avail=74 ack.seq=1270402425 pkt.seq=1270402533 rcv-remain=8117
10:49:20.105844/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=107 avail=74 ack.seq=1270402425 pkt.seq=1270402532 rcv-remain=8117
10:49:20.109087/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=108 avail=74 ack.seq=1270402425 pkt.seq=1270402533 rcv-remain=8117
10:49:20.163817/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=130 avail=74 ack.seq=1270402425 pkt.seq=1270402555 rcv-remain=8117
10:49:20.163901/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=131 avail=74 ack.seq=1270402425 pkt.seq=1270402556 rcv-remain=8117
10:49:20.211894/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=153 avail=74 ack.seq=1270402425 pkt.seq=1270402578 rcv-remain=8117
10:49:20.212092/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=154 avail=74 ack.seq=1270402425 pkt.seq=1270402579 rcv-remain=8117
10:49:20.254187/SRT:RcvQ:worker*E:SRT.c: %470476897:No room to store incoming packet: offset=153 avail=74 ack.seq=1270402425 pkt.seq=1270402578 rcv-remain=8117
So if ffmpeg is not reading the packets quick enough from the SRT stream - how do I get it to work faster? I have tried increasing the CPU power of the instance but that didnt seem to make a difference.. maybe there is some other way?
Although top says that it is running at over %100:
Tasks: 130 total, 1 running, 128 sleeping, 1 stopped, 0 zombie
%Cpu(s): 18.4 us, 0.7 sy, 79.4 ni, 1.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 3852940 total, 3066844 free, 536440 used, 249656 buff/cache
KiB Swap: 4194300 total, 4194300 free, 0 used. 3144320 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1554 root 20 0 1277460 487220 20156 S 196.0 12.6 0:39.01 ffmpeg
1358 nobody 20 0 42740 8332 4944 S 1.0 0.2 0:01.18 nginx
1 root 20 0 38480 6428 3912 S 0.0 0.2 0:02.35 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
Thanks for any advice. James
@jamesrdelaney This error message is usually related to the buffer sizes. See also #703. Default receiver buffer size is approximately 96 Mbits. This buffer should store all the packets within the specified latency, and also have some space for the application to read (FFmpeg).
What is your latency configuration and bitrate? Probably you set latency in Media Gateway.
You might probably need to increase the buffer size, by appending to the following URI
"srt://0.0.0.0:9999?pkt_size=1316&mode=listener"
the following querry values can be added:
sndbuf
- Send buffer size (in bytes)rcvbuf
- Receive buffer size (in bytes)Other FFmpeg SRT URI options can be found here.
I am having a similar problem where Wowza is sending SRT to ffmpeg (so very little parameters on the sending side), and after a random amount of time, I get a similar output. The ack.seq stays the same, and pkt.seq and offset increases. As if it stops acking new packets all together? When it starts, it goes for ever, and no more data is handeled by ffmpeg.
I am testing with a 8mbit/s h264 stream. Sometimes it stops after a few seconds, or even before any video has gone through. Other times, it can go for several minutes. But when this error occurs, everything stops functioning. So cannot really depend on srt :/
ffmpeg is run with ffmpeg -re -i srt://0.0.0.0:9998?mode=listener -map 0:0 -an -c:v copy -f mpegts udp://127.0.0.1:1234
, so it is not really doing a lot with the data. So there should be no bottle necks on the receiving end, ffmpeg is bored, no mentionable cpu usage.
Update: A lot of my traffic was being dropped, as I was going through a VPN with a router that had limited bandwidth support with encrypted vpn.
So I tested a bit, and if you make sure that a significant amount of packets is dropped. This issue can easily be recreated. And it does not recover from it. libsrt/Ffmpeg gives up the whole stream after a while, and exits, which is not in the best interrest of a reliable stream. The posibility for srt to decide to drop packets to get back into sync unlike tcp transports, should be one of the main benefits of srt imho.
I'm still getting this. Gstreamer python refusing to go into play state, then this error repeats. It works and then suddenly doesn't after reloading the command. A reboot of the machine fails to help. I'm doing a local network push. I still do not understand what this buffer setting requirement even means , but I have tried to set it up.
# buffer size 100mb
playbin.set_property("buffer-size", 100 * 1048576)
Removing this has no effect. I can only randomly get it working, so can't replicate how to not get the error. I get one shot at playback. I think the sender and receiver has to be rebooted ?
I now get these stdout messages before failure
16:44:43.293977/SRT:RcvQ:worker D:SRT.t: tsbpd wrap period begins
16:44:43.352669/SRT:TsbPd D:SRT.t: tsbpd wrap period ends
16:44:43.362491/SRT:RcvQ:worker D:SRT.t: tsbpd wrap period begins
@danrossi Messages about TSBPD wrap period are informational. These are not errors. SRT just informs that the timer used to track packets is close to overflow and is ready to be reset. There is nothing you should do about that. It is just to inform about this condition.
Please joint the SRT Alliance Slack channel to get help onthis from the community.
A good point was raised today:
-re
option will slow down the reading:
Read input at native frame rate. Mainly used to simulate a grab device, or live input stream (e.g. when reading from a file). Should not be used with ... live input streams (where it can cause packet loss). By default ffmpeg attempts to read the input(s) as fast as possible. This option will slow down the reading of the input(s) to the native frame rate of the input(s). It is useful for real-time output (e.g. live streaming).
Hi @danrossi @haakonnessjoen @jamesrdelaney @giulianoc Do you guys have updates on this? Was the above note useful?
I'm sorry I had to put my tests off. SRT has been a failure and pushed back for now. I'm using a gstreamer receiver because its programmed via python and in a startup service. So the re option isn't useful for me sorry. I plan to come back to it soon
I've found this ticket and followed the advice in this thread. If someone falls in this issue in the future, I would recomend incresing the "rcvbuf
- Receive buffer size (in bytes)" like @maxsharabayko recommended. I am using GStreamer and this solved the issue for me:
srtsrc uri=srt://myserver.com:7002/?rcvbuf=15000000 ! tsparse ! tsdemux ! h264parse ! appsink name=myapp
In GStreamer the uri
parameters are how we configure the SRT options. The list of available options are listed here:
In my example, I've increased the reciver buffer size to 120 Mbits.
I am trying to receive srt stream using ffplay from pi that has ffmpeg capturing the live stream from a frame grabber attached to pi. To address the problem of "No room to store incoming packet", I have tried increasing the rcvbuf size on ffplay side. But increasing the rcvbuf size to a very large size, allocate more buffer space for incoming packets. So, increasing the rcvbuf size I notice an increase in the packet window (the range of sequence numbers currently buffered or expected). On the sender side, the live stream being sent is, Video: h264, yuv422p(progressive), 1920x1080, q=2-31, 20 fps, 90k tbn and bitrate increasing from 19 Mbps to 20 Mbps. So the only solution I have found for the live streaming the video with mentioned configuration is to use flag -infbuf in ffplay (receiver side) but that adds to the latency too much. Let me know if I can do anything else.
@alri02 : This is a very rare problem that people seem to report once per a time and it's extremely hard to track. The general reason for it is that the receiver application does not read packets fast enough, but which exactly part fails to do its job (could be application, but could be also TSBPD mechanism in SRT), is not exactly clear. We would be glad if you can provide us with an example video where we could be able to check this ourselves (in the meantime we'll be trying ourselves with a video with the above described parameters).
To prevent this for now increasing the receiver buffer may help, but only if you have a temporary spike of quickly sent data, which gets dismissed later. If you have such a problem, for example, that packets get sent with higher speed than it results from the timestamps declared at sending point, then it can't be worked around for now.
When retesting this please also make sure that you use the latest release of SRT.
But I built ffmpeg from source with enable-libsrt flag, shouldn't that pull latest srt release ?
On Thu, Apr 4, 2024 at 12:57 PM Sektor van Skijlen @.***> wrote:
@alri02 https://github.com/alri02 : This is a very rare problem that people seem to report once per a time and it's extremely hard to track. The general reason for it is that the receiver application does not read packets fast enough, but which exactly part fails to do its job (could be application, but could be also TSBPD mechanism in SRT), is not exactly clear. We would be glad if you can provide us with an example video where we could be able to check this ourselves (in the meantime we'll be trying ourselves with a video with the above described parameters).
To prevent this for now increasing the receiver buffer may help, but only if you have a temporary spike of quickly sent data, which gets dismissed later. If you have such a problem, for example, that packets get sent with higher speed than it results from the timestamps declared at sending point, then it can't be worked around for now.
When retesting this please also make sure that you use the latest release of SRT.
— Reply to this email directly, view it on GitHub https://github.com/Haivision/srt/issues/642#issuecomment-2036846982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO6ZL4DNKOGWBNZXE35JDLDY3UWYHAVCNFSM4HFRAPOKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBTGY4DINRZHAZA . You are receiving this because you were mentioned.Message ID: @.***>
That I don't know. Besides, if you built ffmpeg from sources, then you should either have SRT also compiled from sources, or you needed the "devel" SRT packet installed. You can find the SRT release information there.
Hi, I'm using ffmpeg compiled with --enable-libsrt to save a srt feed coming from an Haivision encoder.
The ffmpeg command I'm using generates one ts file every 60 seconds: ffmpeg -i srt://rsis-cdc-mg01:30001 -t 46281 -c:v copy -c:a copy -f segment -segment_list ./liveRecorder.list -segment_time 60 -segment_atclocktime 1 -strftime 1 ./liveRecorder_506757128075%Y-%m-%d%H-%M-%S%s.ts
The source bitrate is 6Mbps.
Sometimes during the day, ffmpeg stopping generating the ts files and exit. Below find the last part of the ffmpeg log:
... 12:22:27.118578/SRT:RcvQ:workerE: SRT.c: %385192064:No room to store incoming packet: offset=8274 avail=6209 ack.seq=8992844 pkt.seq=9001118 rcv-remain=1982 12:22:27.120609/SRT:RcvQ:workerE: SRT.c: %385192064:No room to store incoming packet: offset=8275 avail=6209 ack.seq=8992844 pkt.seq=9001119 rcv-remain=1982 12:22:27.136525/SRT:RcvQ:workerE: SRT.c: %385192064:No room to store incoming packet: offset=8279 avail=6594 ack.seq=8992844 pkt.seq=9001123 rcv-remain=1597 12:22:27.136546/SRT:RcvQ:workerE: SRT.c: %385192064:No room to store incoming packet: offset=8280 avail=6594 ack.seq=8992844 pkt.seq=9001124 rcv-remain=1597 12:22:27.155389/SRT:RcvQ:workerE: SRT.c: %385192064:No room to store incoming packet: offset=8302 avail=7577 ack.seq=8992844 pkt.seq=9001146 rcv-remain=614 12:22:27.155417/SRT:RcvQ:workerE: SRT.c: %385192064:No room to store incoming packet: offset=8303 avail=7577 ack.seq=8992844 pkt.seq=9001147 rcv-remain=614 12:22:27.171317/SRT:RcvQ:worker*E: SRT.c: %385192064:SEQUENCE DISCREPANCY, reception no longer possible. REQUESTING TO CLOSE. [srt @ 0x5577ea452000] Operation not supported: Invalid socket ID: Transport endpoint is not connected. frame=183981 fps= 50 q=-1.0 size=N/A time=01:01:19.84 bitrate=N/A speed=0.998x ^Msrt://rsis-cdc-mg01:30001?rcvbuf=16384: Unknown error occurred frame=183981 fps= 50 q=-1.0 Lsize=N/A time=01:01:19.84 bitrate=N/A speed=0.998x video:2673772kB audio:43122kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Could you please tell me some hints to solve this issue? Best regards Giuliano