Closed sudolee closed 1 year ago
winlinvip
commented
3 years ago What codec is this?
got metadata, width=1024, height=556, vcodec=2, acodec=2
Is it related to your codec? Can you reproduce this issue by changing it to doc/source.flv?
TRANS_BY_GPT3
chundonglinlin
commented
2 years ago There is a problem with the streaming command: ffmpeg -re -i ./1024x556.rmvb -y -f flv rtmp://127.0.0.1/live/test
If ffmpeg does not specify a codec or use the copy option, it will default to streaming with flv1 video codec and mp3 audio codec.
TRANS_BY_GPT3
chundonglinlin
commented
2 years ago Configuration 1: dvr_wait_keyframe on;
Reason for not segmenting: When vcodec=2, which means the codec is H263, if dvr_wait_keyframe is enabled, it will keep waiting for a keyframe of H264. Otherwise, it will not segment and keep writing to the same file continuously. You can refer to this link for more information: https://github.com/ossrs/srs/blob/8bc2759c7e8eaaac95d6e5a5072a03b2378280c3/trunk/src/app/srs_app_dvr.cpp#L866
Configuration 2: dvr_wait_keyframe off;
Reason for incorrect segmentation: This is unrelated to the codec, whether it is H263 or H264. When the first FLV file is closed, and the second FLV file is opened, the duration is not reset to zero. It remains the same as the duration of the previous FLV file. As a result, the newly opened file always meets the segmentation condition.
https://github.com/ossrs/srs/blob/8bc2759c7e8eaaac95d6e5a5072a03b2378280c3/trunk/src/app/srs_app_dvr.cpp#L853
At this point, it will enter into an infinite loop logic, resulting in a Segmentation fault error when writing the DVR file.
TRANS_BY_GPT3
chundonglinlin
commented
2 years ago If streaming is done according to the following configuration, it will enter a DVR infinite loop. The main reason is that when reopening a fragment, the duration of the fragment is not reset to zero.
vhost __defaultVhost__ {
dvr {
enabled on;
dvr_apply all;
dvr_plan segment;
dvr_path ./objs/nginx/html/[app]/[stream].[timestamp].flv;
dvr_duration 10;
dvr_wait_keyframe off;
time_jitter full;
}
}Streaming commands: H264+AAC: ffmpeg -re -i source.200kbps.768x320.flv -c copy -y -f flv rtmp://127.0.0.1/live/stream H263+MP3: ffmpeg -re -i source.200kbps.768x320.flv -y -f flv rtmp://127.0.0.1/live/stream
PS: Configuration for causing an infinite loop is guaranteed to occur.
dvr_plan segment; dvr_wait_keyframe off;
TRANS_BY_GPT3
chundonglinlin
commented
2 years ago winlinvip
commented
2 years ago Minimum reproduction path, setting dvr_wait_keyframe off; will cause problems in the second DVR file.
vhost __defaultVhost__ {
dvr {
enabled on;
dvr_plan segment;
dvr_duration 10;
dvr_wait_keyframe off;
}
ingest livestream {
enabled on;
input {
url ./doc/source.flv;
}
ffmpeg ./objs/ffmpeg/bin/ffmpeg;
engine {
output rtmp://127.0.0.1:[port]/live?vhost=[vhost]/livestream;
}
}
}Around 10 seconds, the second slice starts to refresh the screen, continuously generating new DVR files.
[2022-01-01 12:11:09.130][Trace][45946][wh3t8u2q] dvr stream livestream to file ./objs/nginx/html/live/livestream.1641010269130.flv
[2022-01-01 12:11:09.130][Warn][45946][wh3t8u2q][2] dvr: ignore video error code=3082 : update duration : request sh : dvr video : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/live/livestream.1641010269130.flv
thread [45946][wh3t8u2q]: on_video() [src/app/srs_app_dvr.cpp:835][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:884][errno=2]
thread [45946][wh3t8u2q]: on_dvr_request_sh() [src/app/srs_app_source.cpp:1222][errno=2]
thread [45946][wh3t8u2q]: on_video() [src/app/srs_app_dvr.cpp:835][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:879][errno=2]
thread [45946][wh3t8u2q]: open() [src/app/srs_app_dvr.cpp:79][errno=2]
[2022-01-01 12:11:09.167][Trace][45946][wh3t8u2q] dvr stream livestream to file ./objs/nginx/html/live/livestream.1641010269167.flv
[2022-01-01 12:11:09.167][Warn][45946][wh3t8u2q][2] dvr: ignore audio error code=3082 : update duration : request sh : dvr video : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/live/livestream.1641010269167.flv
thread [45946][wh3t8u2q]: on_audio() [src/app/srs_app_dvr.cpp:820][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:884][errno=2]
thread [45946][wh3t8u2q]: on_dvr_request_sh() [src/app/srs_app_source.cpp:1222][errno=2]
thread [45946][wh3t8u2q]: on_video() [src/app/srs_app_dvr.cpp:835][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:879][errno=2]
thread [45946][wh3t8u2q]: open() [src/app/srs_app_dvr.cpp:79][errno=2]
[2022-01-01 12:11:09.167][Warn][45946][wh3t8u2q][2] dvr: ignore audio error code=3082 : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/live/livestream.1641010269167.flv
thread [45946][wh3t8u2q]: on_audio() [src/app/srs_app_dvr.cpp:820][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:879][errno=2]
thread [45946][wh3t8u2q]: open() [src/app/srs_app_dvr.cpp:79][errno=2]Debugging revealed that update_duration called itself, causing infinite recursion and exhausting the stack, resulting in a crash. Please refer to the following diagram for the specific call stack and function references:
update_duration is responsible for updating the duration of a slice. It has been observed that if the duration exceeds the configured limit, for example, after 10 seconds, the DVR file will be closed and a new DVR file will be created.
Of course, new DVR files will be continuously generated. Except for the first DVR file, the rest will only contain sequence header and metadata if there is no content.
There is an assumption here that is problematic, which is the recursion of update_duration -> close -> open -> on_request_sh -> ... -> update_duration. It is incorrect to trigger update_duration again when on_request_sh requests the sequence header because although audio and video packets are received, they only consist of the sequence header and metadata, and the entire segment does not need to update the duration.
Why does it work fine when waiting for a keyframe? The debugging is shown in the following image:
It can be seen that although update_duration will call on_request_sh and recursively call itself update_duration, the sequence header is considered as a non-sequencer header, so it is considered that the DVR file cannot be closed yet.
The subsequent duration calculation is correct, which is 0, so there is no problem. Therefore, there is an issue here, which is that the duration calculation is incorrect, especially when on_request_sh is recursively called, the duration of the slice is incorrect. It needs to be examined in detail.
`on_request_sh` will call the following function.
SrsDvrSegmentPlan::on_video(SrsSharedPtrMessage* shared_video, SrsFormat* format)
{// When waitkeyframe=on, if the condition is not met, it returns. If waitkeyframe=off, it will recursively call indefinitely.
if ((err = update_duration(shared_video)) != srs_success) {
// Here, the duration of the slice will be recalculated and reset to 0, which is why there is no problem when waitkeyframe=on.
if ((err = SrsDvrPlan::on_video(shared_video, format)) != srs_success) {From here, we can see that there are two issues:1. The calculation of duration is ambiguous, it is not clear which function is responsible for the calculation.2. The process of resetting duration to 0 is very obscure.The root cause of the problem is:1. on_request_sh should not cause the invocation of update_duration, nor should it close DVR slices.2. During on_request_sh, the duration calculation becomes abnormal. The calculation of duration is quite obscure, even if it is calculated correctly.TRANS_BY_GPT3
chundonglinlin
commented
2 years ago Test the 3.0 release version, and this problem also exists. The time when this problem was introduced is approximately Thu Feb 9 15:42:42 2017. Compare: https://github.com/ossrs/srs/compare/455d9b285e82503d89f9b3e9df4e39794f037d54...00abaf4df28b5025215833765b719d1778405bf5
The implementation logic before 3.0 release is as follows:
int SrsDvrSegmentPlan::update_duration(SrsSharedPtrMessage* msg)
{
// update sequence header
if (metadata && (ret = SrsDvrPlan::on_meta_data(metadata)) != ERROR_SUCCESS) {
return ret;
}
if (sh_video && (ret = SrsDvrPlan::on_video(sh_video)) != ERROR_SUCCESS) {
return ret;
}
if (sh_audio && (ret = SrsDvrPlan::on_audio(sh_audio)) != ERROR_SUCCESS) {
return ret;
}There will be no recursive call to update_duration when on_request_sh is called with update_duration. Instead, SrsDvrPlan::on_meta_data, SrsDvrPlan::on_audio, and SrsDvrPlan::on_video will be directly called, followed by on_update_duration to update the segment duration dur. This will skip SrsDvrSegmentPlan::update_duration and directly write the segment.
TRANS_BY_GPT3
chundonglinlin
commented
2 years ago Thank you, Boss Yang, for your guidance. The analysis is very clear. Currently, I have thought of several repair methods:
Method 1: Clean up fragment duration when open/close;
Method 2: Exclude sh from duration calculation when updating duration;
Method 3: Add a variable flag in on_request_sh to block the call to update_duration;
Method 4: Refer to the handling logic in the 2.0 release.
TRANS_BY_GPT3
winlinvip
commented
2 years ago Solving the infinite recursion problem: https://github.com/ossrs/srs/commit/4d09b8caae8e39a3b7eed4e0f71a65c6dfc6387f
After trying it out, the problem of continuous slicing no longer occurs, and it has been resolved. However, the duration issue has not been completely resolved and still poses a potential risk.
TRANS_BY_GPT3
zhangzhenglava
commented
2 years ago Using Weibo Vision 5G terminal to push stream to SRS 4.0.249, if RTC is configured, the same server error code=3082 error code will appear. The problem exhibits the same symptoms; commenting out the RTC configuration restores normal operation.
Terminal settings:
SRS 4.0 configuration:
listen 1949;
max_connections 1000;
srs_log_tank file;
srs_log_file ./objs/srs.log;
daemon on;
http_api {
enabled on;
listen 1985;
raw_api {
enabled on;
allow_reload on;
allow_query on;
allow_update on;
}
}
stats {
network 0;
disk sda sdb xvda xvdb;
}
http_server { enabled on; listen 8817; dir ./objs/nginx/html; } vhost defaultVhost {
# enabled on;
# rtmp_to_rtc on;
# rtc_to_rtmp on;
#}
tcp_nodelay on min_latency on;
play {
gop_cache off;
queue_length 10;
mw_latency 100;
}
publish {
mr off;
mr_latency 350;
}
hls {
enabled on;
hls_fragment 0.2;
hls_window 2;
hls_wait_keyframe off;
}
http_remux {
enabled on;
mount [vhost]/[app]/[stream].flv;
}
dvr {
enabled on;
dvr_path /usr/sdb/srs-video/[app]/[stream]/[2006][01][02]/[2006][01][02][15][04].mp4;
dvr_plan session;
dvr_wait_keyframe on;
time_jitter full;
dvr_apply all;
}
http_hooks {
enabled on;
on_dvr http://localhost:8087/srs/receive;
on_connect http://localhost:8087/srs/receive;
on_close http://localhost:8087/srs/receive;
on_publish http://localhost:8087/srs/receive;
on_unpublish http://localhost:8087/srs/receive;
}}
Console error message:
[2022-03-17 16:39:50.797][Error][22038][w9232kw0][0] serve error code=3082 : service cycle : rtmp: stream service : hub publish : dvr publish : publish : open segment : DVR can't append to exists path=/usr/sdb/srs-video/ry_live/M300002/20220317/202203171639.mp4
thread [22038][w9232kw0]: do_cycle() [src/app/srs_app_rtmp_conn.cpp:217][errno=0]
thread [22038][w9232kw0]: service_cycle() [src/app/srs_app_rtmp_conn.cpp:414][errno=0]
thread [22038][w9232kw0]: on_publish() [src/app/srs_app_source.cpp:2507][errno=0]
thread [22038][w9232kw0]: on_publish() [src/app/srs_app_source.cpp:1143][errno=0]
thread [22038][w9232kw0]: on_publish() [src/app/srs_app_dvr.cpp:973][errno=0]
thread [22038][w9232kw0]: on_publish() [src/app/srs_app_dvr.cpp:716][errno=0]
thread [22038][w9232kw0]: open() [src/app/srs_app_dvr.cpp:79][errno=0]
DVR recording:
TRANS_BY_GPT3
winlinvip
commented
1 year ago Should be fixed in SRS 4
Description'
Please ensure that you maintain the markdown structure.
SRS version (Version):
4.0.177The log of SRS is as follows (Log): When dvr_wait_keyframe is enabled, there is no segmentation and no abnormal log. When dvr_wait_keyframe is disabled, there is segmentation, but the segmentation is incorrect. The log is as follows:
The configuration of SRS is as follows (Config):
http_server { enabled on; listen 8080; dir ./objs/nginx/html; }
http_api { enabled on; listen 1985; } stats { network 0; }
rtc_server { enabled on; listen 8000;
@see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#config-candidate
}
vhost defaultVhost { rtc { enabled on;
@see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtmp-to-rtc
}