search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://airensoft.com/ome.html
GNU Affero General Public License v3.0
2.58k stars 1.06k forks source link

Ever increasing stream delta #742

Closed deepsurvey closed 2 years ago

deepsurvey commented 2 years ago

I am running an origin edge configuration. At the origin I see for each feed the stream delta increasing all the time. Does anyone know why this happens? 

origin_1  | [2022-04-22 13:55:26.387] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:23   | OvenMediaEngine v0.13.2 () is started on [044578540b43] (Linux x86_64 - 5.13.0-39-generic, #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022)

origin_1  | [2022-04-22 07:06:49.084] I [AW-LLDASH0:44] LLDASH.Stat | cmaf_packetizer.cpp:883  | [#default#volans/sb] Jitter statistics
origin_1  |   - Elapsed: 155061ms (current: 1650611209084ms, start: 1650611054023ms)
origin_1  |   - Jitter: -22ms (-22ms - 0ms), correction: 0ms => 0ms (0ms)
origin_1  |   - Stream delta: 155083ms
origin_1  |     - Video: last PTS: 155083ms, start: 0ms, delta: 155083ms
top at origin looks good:
top - 09:21:55 up 7 days,  1:46,  2 users,  load average: 0,75, 0,48, 0,39
Threads:  94 total,   0 running,  94 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2,0 us,  1,3 sy,  0,0 ni, 96,5 id,  0,0 wa,  0,0 hi,  0,2 si,  0,0 st
MiB Mem :  64026,7 total,  60166,9 free,   1843,9 used,   2015,9 buff/cache
MiB Swap:   2048,0 total,   1796,5 free,    251,5 used.  61694,5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                
 132907 root      20   0 7062680 154972  15544 S   3,0   0,2   0:33.77 AW-WebRTC0                                                                                                             
 132906 root      20   0 7062680 154972  15544 S   1,3   0,2   0:13.42 OutboundWorker                                                                                                         
 132905 root      20   0 7062680 154972  15544 S   1,0   0,2   0:12.05 InboundWorker                                                                                                          
 132908 root      20   0 7062680 154972  15544 S   1,0   0,2   0:10.42 AW-HLS0                                                                                                                
 132909 root      20   0 7062680 154972  15544 S   0,7   0,2   0:08.83 AW-DASH0                                                                                                               
 132911 root      20   0 7062680 154972  15544 S   0,7   0,2   0:08.66 AW-OVT0                                                                                                                
 132915 root      20   0 7062680 154972  15544 S   0,7   0,2   0:05.62 StreamMotor                                                                                                            
 132933 root      20   0 7062680 154972  15544 S   0,7   0,2   0:09.17 StreamMotor                                                                                                            
 132935 root      20   0 7062680 154972  15544 S   0,7   0,2   0:02.91 StreamWorker                                                                                                           
 132936 root      20   0 7062680 154972  15544 S   0,7   0,2   0:02.92 StreamWorker                                                                                                           
 132950 root      20   0 7062680 154972  15544 S   0,7   0,2   0:09.56 StreamMotor                                                                                                            
 132951 root      20   0 7062680 154972  15544 S   0,7   0,2   0:03.09 StreamWorker                                                                                                           
 132961 root      20   0 7062680 154972  15544 S   0,7   0,2   0:02.46 StreamWorker                                                                                                           
 132901 root      20   0 7062680 154972  15544 S   0,3   0,2   0:01.80 SPSRT-T9999                                                                                                            
 132910 root      20   0 7062680 154972  15544 S   0,3   0,2   0:05.28 AW-LLDASH0                                                                                                             
 132918 root      20   0 7062680 154972  15544 S   0,3   0,2   0:02.02 StreamWorker                                                                                                           
 132919 root      20   0 7062680 154972  15544 S   0,3   0,2   0:02.00 StreamWorker                                                                                                           
 132920 root      20   0 7062680 154972  15544 S   0,3   0,2   0:01.99 StreamWorker                                                                                                           
 132921 root      20   0 7062680 154972  15544 S   0,3   0,2   0:01.97 StreamWorker                                                                                                           
 132922 root      20   0 7062680 154972  15544 S   0,3   0,2   0:01.97 StreamWorker                                                                                                           
 132923 root      20   0 7062680 154972  15544 S   0,3   0,2   0:02.15 StreamWorker                                                                                                           
 132925 root      20   0 7062680 154972  15544 S   0,3   0,2   0:00.26 StreamWorker                                                                                                           
 132934 root      20   0 7062680 154972  15544 S   0,3   0,2   0:02.91 StreamWorker                                                                                                           
 132938 root      20   0 7062680 154972  15544 S   0,3   0,2   0:02.87 StreamWorker                                                                                                           
 132939 root      20   0 7062680 154972  15544 S   0,3   0,2   0:02.84 StreamWorker                                                                                                           
 132941 root      20   0 7062680 154972  15544 S   0,3   0,2   0:03.04 StreamWorker                                                                                                           
 132952 root      20   0 7062680 154972  15544 S   0,3   0,2   0:03.06 StreamWorker                                                                                                           
 132954 root      20   0 7062680 154972  15544 S   0,3   0,2   0:03.06 StreamWorker                                                                                                           
 132955 root      20   0 7062680 154972  15544 S   0,3   0,2   0:03.01 StreamWorker                                                                                                           
 132956 root      20   0 7062680 154972  15544 S   0,3   0,2   0:02.96 StreamWorker 
origin_1  | [2022-04-22 14:35:42.510] I [AW-LLDASH0:55] LLDASH.Stat | cmaf_packetizer.cpp:883  | [#default#volans/ps] Jitter statistics
origin_1  |   - Elapsed: 2410290ms (current: 1650638142510ms, start: 1650635732220ms)
origin_1  |   - Jitter: 13ms (13ms - 0ms), correction: 0ms => 0ms (0ms)
origin_1  |   - Stream delta: 2410277ms
origin_1  |     - Video: last PTS: 2410277ms, start: 0ms, delta: 2410277ms
origin_1  | [2022-04-22 14:35:44.171] W [SPOvtPub-T9000:27] Socket | socket.cpp:1977 | [#50] [0x7fc0f80008e0] Socket was closed even though there was 206 commands in the socket
origin_1  | [2022-04-22 14:35:44.172] I [SPOvtPub-T9000:27] OVT | ovt_publisher.cpp:225  | OvtProvider is disconnected : <ClientSocket: 0x7fc0f80008e0, #50, Closed, TCP, Nonblocking, 172.26.96.18:46624>
origin_1  | [2022-04-22 14:35:46.022] I [AW-LLDASH1:56] LLDASH.Stat | cmaf_packetizer.cpp:883  | [#default#volans/inno] Jitter statistics
origin_1  |   - Elapsed: 232109ms (current: 1650638146022ms, start: 1650637913913ms)
origin_1  |   - Jitter: 1209ms (1209ms - 0ms), correction: 0ms => 0ms (0ms)
origin_1  |   - Stream delta: 230900ms
origin_1  |     - Video: last PTS: 230900ms, start: 0ms, delta: 230900ms
origin_1  | [2022-04-22 14:35:47.974] I [SPOvtPub-T9000:27] OVT | ovt_publisher.cpp:225  | OvtProvider is disconnected : <ClientSocket: 0x7fc0f8017ff0, #38, Closed, TCP, Nonblocking, 172.26.96.18:46580>
origin_1  | [2022-04-22 14:35:47.974] W [SPOvtPub-T9000:27] Socket | socket.cpp:1977 | [#38] [0x7fc0f8017ff0] Socket was closed even though there was 8 commands in the socket
origin_1  | [2022-04-22 14:35:48.448] I [AW-LLDASH1:56] LLDASH.Stat | cmaf_packetizer.cpp:883  | [#default#volans/fwd] Jitter statistics
origin_1  |   - Elapsed: 265997ms (current: 1650638148448ms, start: 1650637882451ms)

Weird is that I can have a SSH connection open for days on end without loosing socket connection to that origin.

dimiden commented 2 years ago

@deepsurvey Because Stream delta is calculated as <Last frame's PTS> - <First frame's PTS>, it's normal to continue increasing. Usually, a value close to Elapsed which means <Last frame received time> - <First frame received time>.

This is simply a log that is output to show statistics, and does not mean an error. (I think this term may be confusing to someone other than me, so I'll consider changing it.) If you think that the log is unnecessary, I recommend that you add the following settings in Logger.xml.

<Tag name=".*\.Stat" level="warn" />
deepsurvey commented 2 years ago

Hello dimiden,

Thanks for the response. The problem I am experiencing is that my viewers have an ever increasing delay. After a few hours streaming they lag 30 minutes in feed... and I can not find out who is buffering. I never had this issue in the past. We use OME for remote controlling operations near real time.. and somehow I have lost the low latency part ;-P

Do you have any Idea's how I can find out why this happens. I am RTSP pulling camera feeds into an Origin server and all viewer connect to an Edge server. OVT between the origin and edge.

getroot commented 2 years ago

@deepsurvey

  1. Is there a 30-minute delay after the player disconnects and reconnects?
  2. What is the CPU usage per thread of Edge?
  3. Isn't there anything strange in Edge's log? (Like the queue keeps getting bigger..)
  4. Did you turn on encoding in Origin?
deepsurvey commented 2 years ago

Please see the answers below. Yesterday I did recompile both the Edge and Origin with the release version of the GIT and the results are the same. We started streaming at 08:00 and at say 11:00the played feed was lagging 1.5 hours behind the source.

Is there a 30-minute delay after the player disconnects and reconnects? yes

  1. I can not find any weird stuff in the edge log.

What is the CPU usage per thread of Edge?

No weird high load here.

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                            
3090241 root      20   0 4926624  28736  15288 S  35.3   0.1   3:37.73 OvenMediaEngine 
top - 11:54:48 up 37 days, 19:44,  3 users,  load average: 0.42, 0.25, 0.20
Threads:  94 total,   0 running,  94 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.8 us,  1.4 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :  31360.8 total,   6995.2 free,   2209.2 used,  22156.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  28790.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                            
3090347 root      20   0 4926624  28736  15288 S   3.0   0.1   0:24.63 AW-WebRTC0                                                                                         
3090346 root      20   0 4926624  28736  15288 S   1.3   0.1   0:10.19 OutboundWorker                                                                                     
3090349 root      20   0 4926624  28736  15288 S   1.0   0.1   0:06.79 AW-MPEGTSPush0                                                                                     
3101001 root      20   0 4926624  28736  15288 S   1.0   0.1   0:01.14 StreamMotor                                                                                        
3090319 root      20   0 4926624  28736  15288 S   0.7   0.1   0:07.58 SPICE-U10000                                                                                       
3090321 root      20   0 4926624  28736  15288 S   0.7   0.1   0:07.65 SPICE-U10002                                                                                       
3090348 root      20   0 4926624  28736  15288 S   0.7   0.1   0:06.74 AW-File0                                                                                           
3090350 root      20   0 4926624  28736  15288 S   0.7   0.1   0:06.42 AW-RTMPPush0                                                                                       
3100965 root      20   0 4926624  28736  15288 S   0.7   0.1   0:01.10 StreamMotor                                                                                        
3100969 root      20   0 4926624  28736  15288 S   0.7   0.1   0:01.00 StreamWorker                                                                                       
3101006 root      20   0 4926624  28736  15288 S   0.7   0.1   0:01.16 StreamWorker                                                                                       
3101027 root      20   0 4926624  28736  15288 S   0.7   0.1   0:00.14 StreamWorker                                                                                       
3101031 root      20   0 4926624  28736  15288 S   0.7   0.1   0:01.07 StreamMotor                                                                                        
3101032 root      20   0 4926624  28736  15288 S   0.7   0.1   0:00.37 StreamWorker                                                                                       
3101038 root      20   0 4926624  28736  15288 S   0.7   0.1   0:01.10 StreamWorker                                                                                       
3090313 root      20   0 4926624  28736  15288 S   0.3   0.1   0:03.09 SPRtcSig-T3334                                                                                     
3090316 root      20   0 4926624  28736  15288 S   0.3   0.1   0:20.73 SPRtcSig-T5556                                                                                     
3090320 root      20   0 4926624  28736  15288 S   0.3   0.1   0:06.73 SPICE-U10001                                                                                       
3090324 root      20   0 4926624  28736  15288 S   0.3   0.1   0:05.82 SPICE-U10005                                                                                       
3100966 root      20   0 4926624  28736  15288 S   0.3   0.1   0:00.31 StreamWorker                                                                                       
3100968 root      20   0 4926624  28736  15288 S   0.3   0.1   0:00.26 StreamWorker                                                                                       
3100970 root      20   0 4926624  28736  15288 S   0.3   0.1   0:00.27 StreamWorker                                                                                       
3100971 root      20   0 4926624  28736  15288 S   0.3   0.1   0:00.27 StreamWorker                                                                                       
3100974 root      20   0 4926624  28736  15288 S   0.3   0.1   0:00.18 StreamWorker

Did you turn on encoding in Origin?

Not that I am aware of. As far as I know I just pass the H264 feeds from camera to ws.

<?xml version="1.0" encoding="UTF-8" ?>

<Server version="8">
        <Name>OvenMediaEngine</Name>
        <!-- Host type (origin/edge) -->
        <Type>origin</Type>
        <!-- Specify IP address to bind (* means all IPs) -->
        <IP>*</IP>
        <PrivacyProtection>false</PrivacyProtection>
        <!--
        To get the public IP address(mapped address of stun) of the local server.
        This is useful when OME cannot obtain a public IP from an interface, such as AWS or docker environment.
        If this is successful, you can use ${PublicIP} in your settings.
        -->
        <!--<StunServer>stun.l.google.com:19302</StunServer>-->

        <!-- Settings for the ports to bind -->
        <Bind>
                <Providers>
                        <!-- Pull providers -->
                        <RTSPC>
                                <WorkerCount>1</WorkerCount>
                        </RTSPC>
                        <OVT>
                                <WorkerCount>1</WorkerCount>
                        </OVT>
                        <!-- Push providers -->
                        <RTMP>
                                <Port>${env:OME_RTMP_PROV_PORT:1935}</Port>
                                <WorkerCount>1</WorkerCount>
                        </RTMP>
                        <SRT>
                                <Port>${env:OME_SRT_PROV_PORT:9999}</Port>
                                <WorkerCount>1</WorkerCount>
                        </SRT>
                        <MPEGTS>
                                <!--
                                        Listen on port 4000-4005
                                        This is just a demonstration to show that you can configure the port in several ways
                                -->
                                <Port>${env:OME_MPEGTS_PROV_PORT:4000/udp}</Port>
                        </MPEGTS>
                       <WebRTC>
                                <Signalling>
                                        <Port>${env:OME_SIGNALLING_PORT:3333}</Port>
                                        <WorkerCount>1</WorkerCount>
                                        <!-- If you want to use TLS, specify the TLS port -->
                                        <!-- <TLSPort>3334</TLSPort> -->
                                </Signalling>
                                <IceCandidates>
                                        <TcpRelay>${env:OME_TCP_RELAY_ADDRESS:*:3478}</TcpRelay>
                                        <TcpForce>true</TcpForce>
                                        <TcpRelayWorkerCount>1</TcpRelayWorkerCount>
                                        <IceCandidate>${env:OME_ICE_CANDIDATES:*:10006/udp}</IceCandidate>
                                </IceCandidates>
                        </WebRTC>
                </Providers>

                <Publishers>
                        <!-- The OVT is protocol for ORIGIN-EDGE -->
                        <OVT>
                                <Port>${env:OME_ORIGIN_PORT:9000}</Port>
                                <WorkerCount>1</WorkerCount>
                        </OVT>
                        <HLS>
                                <Port>${env:OME_HLS_STREAM_PORT:8080}</Port>
                                <WorkerCount>1</WorkerCount>
                                <!-- If you want to use TLS, specify the TLS port -->
                                <!-- <TLSPort>443</TLSPort> -->
                        </HLS>
                        <DASH>
                                <Port>${env:OME_DASH_STREAM_PORT:8080}</Port>
                                <WorkerCount>1</WorkerCount>
                                <!-- If you want to use TLS, specify the TLS port -->
                                <!-- <TLSPort>443</TLSPort> -->
                        </DASH>
                        <WebRTC>
                                <Signalling>
                                        <Port>${env:OME_SIGNALLING_PORT:3333}</Port>
                                        <WorkerCount>1</WorkerCount>
                                        <!-- If you want to use TLS, specify the TLS port -->
                                        <!-- <TLSPort>3334</TLSPort> -->
                                </Signalling>
                                <IceCandidates>
                                        <TcpRelay>${env:OME_TCP_RELAY_ADDRESS:*:3478}</TcpRelay>
                                        <TcpForce>true</TcpForce>
                                        <TcpRelayWorkerCount>1</TcpRelayWorkerCount>
                                        <IceCandidate>${env:OME_ICE_CANDIDATES:*:10006/udp}</IceCandidate>
                                </IceCandidates>
                        </WebRTC>
                </Publishers>
        </Bind>
        <VirtualHosts>
                <!--
                        You can include multiple XML files by doing the following:
                        <VirtualHost include="sites-enabled/*.xml" />
                -->
                <VirtualHost include="VHost*.xml" />
                <VirtualHost>
                        <Name>default</Name>

                        <!-- Settings for multi ip/domain and TLS -->
                        <Host>
                                <Names>
                                        <!-- Host names
                                                <Name>stream1.airensoft.com</Name>
                                                <Name>stream2.airensoft.com</Name>
                                               <Name>*.sub.airensoft.com</Name>
                                                <Name>192.168.0.1</Name>
                                        -->
                                        <Name>*</Name>
                                </Names>
                                <!--
                                <TLS>
                                        <CertPath>path/to/file.crt</CertPath>
                                        <KeyPath>path/to/file.key</KeyPath>
                                        <ChainCertPath>path/to/file.crt</ChainCertPath>
                                </TLS>
                                -->
                        </Host>
                        <Origins>
                                <Origin>
                                        <Location>/${env:APPLICATION_NAME}/${env:CAMERA_1}</Location> <!--/app_name/rtsp_stream_name-->
                                                <Pass>
                                                        <Scheme>rtsp</Scheme>
                                                        <Urls><Url>${env:CAMERA_1_RTSP_URL}</Url></Urls>
                                                        <ForwardQueryParams>true</ForwardQueryParams>
                                                </Pass>
                                </Origin>
                                <Origin>
                                        <Location>/${env:APPLICATION_NAME}/${env:CAMERA_2}</Location> <!--/app_name/rtsp_stream_name-->
                                                <Pass>
                                                        <Scheme>rtsp</Scheme>
                                                        <Urls><Url>${env:CAMERA_2_RTSP_URL}</Url></Urls>
                                                </Pass>
                                </Origin>
                                <Origin>
                                        <Location>/${env:APPLICATION_NAME}/${env:CAMERA_3}</Location> <!--/app_name/rtsp_stream_name-->
                                                <Pass>
                                                        <Scheme>rtsp</Scheme>
                                                        <Urls><Url>${env:CAMERA_3_RTSP_URL}</Url></Urls>
                                                </Pass>
                                </Origin>
                       </Origins>
                        <!-- Settings for applications -->
                        <Applications>
                                <Application>
                                        <Name>${env:APPLICATION_NAME}</Name>
                                        <!-- Application type (live/vod) -->
                                        <Type>live</Type>
                                        <OutputProfiles>
                                                <OutputProfile>
                                                        <Name>bypass_stream</Name>
                                                        <OutputStreamName>${OriginStreamName}</OutputStreamName>
                                                        <Encodes>
                                                                <Audio>
                                                                        <Bypass>true</Bypass>
                                                                </Audio>
                                                                <Video>
                                                                        <Bypass>true</Bypass>
                                                                </Video>
                                                                <Audio>
                                                                        <Codec>opus</Codec>
                                                                        <Bitrate>128000</Bitrate>
                                                                        <Samplerate>48000</Samplerate>
                                                                        <Channel>2</Channel>
                                                               </Audio>
                                                        </Encodes>
                                                </OutputProfile>
                                        </OutputProfiles>
                                        <Providers>
                                                <OVT />
                                                <WebRTC />
                                                <RTMP />
                                                <SRT />
                                                <RTSPPull />
                                                <MPEGTS>
                                                        <StreamMap>
                                                                <!--
                                                                        Set the stream name of the client connected to the port to "stream_${Port}"
                                                                        For example, if a client connets to port 4000, OME creates a "stream_4000" stream
                                                                -->
                                                                <Stream>
                                                                        <Name>stream_${Port}</Name>
                                                                        <Port>4000</Port>
                                                                </Stream>
                                                                <!--
                                                                <Stream>
                                                                        <Name>stream_4005</Name>
                                                                        <Port>4005</Port>
                                                                </Stream> -->
                                                        </StreamMap>
                                                </MPEGTS>
                                        </Providers>
                                        <Publishers>
                                                <AppWorkerCount>4</AppWorkerCount>
                                                <StreamWorkerCount>1</StreamWorkerCount>
                                                <OVT />
                                                <WebRTC>
                                                        <Timeout>30000</Timeout>
                                                        <Rtx>false</Rtx>
                                                        <Ulpfec>true</Ulpfec>
                                                        <JitterBuffer>false</JitterBuffer>
                                                </WebRTC>
                                                <HLS>
                                                        <SegmentDuration>5</SegmentDuration>
                                                        <SegmentCount>3</SegmentCount>
                                                        <CrossDomains>
                                                                <Url>*</Url>
                                                        </CrossDomains>
                                                </HLS>
                                                <DASH>
                                                        <SegmentDuration>5</SegmentDuration>
                                                        <SegmentCount>3</SegmentCount>
                                                        <CrossDomains>
                                                                <Url>*</Url>
                                                        </CrossDomains>
                                                </DASH>
                                               <LLDASH>
                                                        <SegmentDuration>5</SegmentDuration>
                                                        <CrossDomains>
                                                                <Url>*</Url>
                                                        </CrossDomains>
                                                </LLDASH>
                                        </Publishers>
                                </Application>
                        </Applications>
                </VirtualHost>
        </VirtualHosts>
</Server>
getroot commented 2 years ago

If OME is the cause of the 30 minute delay, the stream data should be buffered in OME's queue. If so, the memory usage will be very high. However, the memory usage rate in the process information you showed is very low (0.1%).

Could it be that the stream is buffered on your encoder side?

deepsurvey commented 2 years ago

Hi,

The encoder side, would be multiple an Axis webcams. If I connect to the live RTSP feed using VLC, that feed is near real time.

My config is the following:

RTSP H624 source <-- Origin OME <-- OVT over LTE network (4G Mobile) <-- Edge OME in data centre <-- broadband connected OME Players in website (Chrome as Browser) Just to put it into perspective each Origin is only serving 4-6 video feeds and we are only running 3 live vessels at the moment, so the edge handling 24 feeds max.. so in numbers and load, that should be no problem. Each Origin is a Intel NUC gen 13 i7 16Gig RAM.

Normally if the LTE connection is crappy, the feed would stutter, and that is fully acceptable, for we need an as low latency feed as possible.

Can I limit the OME queue depth?
Is there way to "see" if a queue is building?

I can see a Time field in one of my feeds and I do observe the following: At the RTSP feed in VLC I see seconds tick "normally" At the OME I see them tick in slow-motion, but the feed does not skip any second... so the delay is gradually building up over time. If I execute sudo docker-compose restart or have someone unplug and replug the network cable of the origin OME, the players will obviously loose connection and regain it, and the Feed will jump to present time and the delay game starts again.

Volans OME

Volans time delay

I just hope I am missing something incredibly stupid here...

Fun fact I have an other Vessel working in the same area's that does not experience this problem. (same edge server used to access that origin).

I really love the prompt replies and great support you and your team are providing, thanks again.

getroot commented 2 years ago

The surest way to see if OME's queue continues to grow is to observe OME's memory utilization.

As you said the feed doesn't skip any seconds, someone must be keeping it.

If so, check which of [RTSP Source] , [OME Origin] , [OME Edge] , and [Player] continues to increase in memory usage.

What I suspect the most in the situation is that the network from [RTSP Source] to [OME Origin] is not fast enough, so frames keep piling up in the RTSP Source.

deepsurvey commented 2 years ago

What I suspect the most in the situation is that the network from [RTSP Source] to [OME Origin] is not fast enough, so frames keep piling up in the RTSP Source.

I will investigate, but then I do not understand how I am able to play the RTSP source at VLC from our office and that one seems fine. That should illuminate the RTSP source. Or it means the network connection of the OME origin is crappy. Seems ok.

Settings for eno1:

    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: on (auto)
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

I will monitor the Origin over some time to see if memory usages grows.

# watch -n 5 free -m

Every 5,0s: free -m                                                                                                                       volans: Wed Apr 27 10:19:09 2022
sh: 0: getcwd() failed: No such file or directory
              total        used        free      shared  buff/cache   available
Mem:          64026        2985       56972           8        4068       60582
Swap:          2047         281        1766

              total        used        free      shared  buff/cache   available
Mem:          64026        4373       55550           8        4102       59191
Swap:          2047         281        1766

after 2 hours running:

available mem at origin decreased to: 59191 -> 1.4Gig
At the Edge -> 28749 to 28726 .. nothing funny there.

Restarted the Origin:
              total        used        free      shared  buff/cache   available
Mem:          64026        1710       58213           8        4103       61854
Swap:          2047         281        1766

OK... memory back up to almost 62Gigs free mem space, so it is the origin that is queueing

Question? Does the origin queue at the RTSP ingesting side or at the OVT "output side"?

top - 10:16:21 up 12 days,  2:40,  3 users,  load average: 0,33, 0,19, 0,17
Threads:  72 total,   0 running,  72 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1,3 us,  0,7 sy,  0,0 ni, 97,7 id,  0,0 wa,  0,0 hi,  0,2 si,  0,0 st
MiB Mem :  64026,7 total,  56976,9 free,   2982,4 used,   4067,4 buff/cache
MiB Swap:   2048,0 total,   1766,6 free,    281,4 used.  60585,6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                            
 308141 root      20   0 6387596   1,3g  15836 S   1,0   2,0   0:59.63 InboundWorker                                                                                      
 308143 root      20   0 6387596   1,3g  15836 S   1,0   2,0   1:36.13 OutboundWorker                                                                                     
 308147 root      20   0 6387596   1,3g  15836 S   1,0   2,0   1:42.84 AW-WebRTC2                                                                                         
 308151 root      20   0 6387596   1,3g  15836 S   0,7   2,0   1:04.54 AW-HLS2                                                                                            
 318980 root      20   0 6387596   1,3g  15836 S   0,7   2,0   0:42.11 StreamMotor                                                                                        
 318983 root      20   0 6387596   1,3g  15836 S   0,7   2,0   0:36.70 StreamMotor                                                                                        
 308137 root      20   0 6387596   1,3g  15836 S   0,3   2,0   1:27.36 SPSRT-T9999
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.