search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://airensoft.com/ome.html
GNU Affero General Public License v3.0
2.58k stars 1.06k forks source link

SIGSEGV interrupt after TLS warning without having TLS enabled #1721

Open bartvp1 opened 2 weeks ago

bartvp1 commented 2 weeks ago

Describe the bug Even though I dont use TLS in my OvenMediaEngine configuration, it still occasionally warns about not being able to do something related to TLS, which often coincides with a crash of the Origin server.

I can't confirm whether this is related to the TLS warning, but since most crashes end with these same warnings I have my suspicions.

To Reproduce No steps to reproduce, this seems to happen out of the blue

Expected behavior I would expect there not to be any logs or crashes related to TLS if I'm not using any TLS for this service, because the load balancer terminates it.

Logs Origin:

[2024-10-15 08:19:33.916] W [SPRTMP-t1935:11] OpenSSL | tls.cpp:328  | Tls::Read() returns 5 (errno: 11): [OpenSSL] error:00000000:lib(0)::reason(0) (0)
[2024-10-15 08:19:33.916] W [SPAPISvr-t8080:20] OpenSSL | tls.cpp:328  | Tls::Read() returns 5 (errno: 11): [OpenSSL] error:00000000:lib(0)::reason(0) (0)
[2024-10-15 08:19:33.916] C [SPRTMP-t1935:11] OvenMediaEngine | signals.cpp:124  | OME v0.16.8 ((From archive)) received signal 11 (SIGSEGV), interrupt.
[2024-10-15 08:19:33.916] C [SPAPISvr-t8080:20] OvenMediaEngine | signals.cpp:124  | OME v0.16.8 ((From archive)) received signal 11 (SIGSEGV), interrupt.

Server

dimiden commented 2 weeks ago

Since there is no need to use TLS in the SPRTMP thread, the related logs should not be output. It seems unusual that they are being printed.

It is likely that an issue such as a stack overflow may have caused incorrect execution. For a more detailed analysis, please share the full logs, Server.xml, and any dump files generated in the /usr/share/ovenmediaengine/dumps directory inside the Docker container.

bartvp1 commented 2 weeks ago

Here's the server configuration for the origin:

<?xml version="1.0" encoding="UTF-8"?>

<Server version="8">
    <Name>OvenMediaEngine</Name>
    <Type>origin</Type>
    <IP>*</IP>
    <StunServer>stun.l.google.com:19302</StunServer>

    <Managers>
        <Host>
            <Names>
                <Name>*</Name>
            </Names>
        </Host>
        <API>
            <AccessToken>${env:OME_API_ACCESS_TOKEN}</AccessToken>
        </API>
    </Managers>

    <Bind>
        <Managers>
            <API>
                <Port>8080</Port>
            </API>
        </Managers>

        <Providers>
            <RTMP>
                <Port>1935</Port>
            </RTMP>
        </Providers>

        <Publishers>
            <OVT>
                <Port>9000</Port>
            </OVT>
        </Publishers>
    </Bind>

    <VirtualHosts>
        <VirtualHost include="VHost*.xml" />
        <VirtualHost>
            <Name>default</Name>
            <Host>
                <Names>
                    <Name>*</Name>
                </Names>
            </Host>

            <AdmissionWebhooks>
                <ControlServerUrl>${env:ADMISSION_ADDRESS}</ControlServerUrl>
                <SecretKey>${env:ADMISSION_SECRET_KEY}</SecretKey>
                <Timeout>3000</Timeout>
                <Enables>
                    <Providers>rtmp</Providers>
                </Enables>
            </AdmissionWebhooks>

            <OriginMapStore>
                <RedisServer>
                    <Host>${env:REDIS_HOST}:${env:REDIS_PORT}</Host>
                    <Auth>${env:REDIS_PASSWORD}</Auth>
                </RedisServer>

                <OriginHostName>${env:POD_IP}</OriginHostName>
            </OriginMapStore>

            <Applications>
                <Application>
                    <Name>app</Name>
                    <Type>live</Type>

                    <OutputProfiles>
                        <OutputProfile>
                            <Name>ABR</Name>
                            <OutputStreamName>${OriginStreamName}</OutputStreamName>
                            <Playlist>
                                <Name>abr</Name>
                                <FileName>abr</FileName>
                                <Options>
                                    <WebRtcAutoAbr>true</WebRtcAutoAbr>
                                </Options>

                                <Rendition>
                                    <Name>Full HD</Name>
                                    <Video>video_1080</Video>
                                    <Audio>cond_audio_opus</Audio>
                                </Rendition>
                                <Rendition>
                                    <Name>HD</Name>
                                    <Video>video_720</Video>
                                    <Audio>cond_audio_opus</Audio>
                                </Rendition>
                                <Rendition>
                                    <Name>SD</Name>
                                    <Video>video_360</Video>
                                    <Audio>cond_audio_opus</Audio>
                                </Rendition>
                            </Playlist>

                            <Encodes>
                                <Video>
                                    <Name>video_1080</Name>
                                    <Codec>h264</Codec>
                                    <Height>1080</Height>
                                    <Bitrate>2048000</Bitrate>
                                    <Framerate>24</Framerate>
                                    <ThreadCount>4</ThreadCount>
                                    <Preset>slower</Preset>
                                </Video>
                                <Video>
                                    <Name>video_720</Name>
                                    <Codec>h264</Codec>
                                    <Height>720</Height>
                                    <Bitrate>1024000</Bitrate>
                                    <Framerate>24</Framerate>
                                    <ThreadCount>4</ThreadCount>
                                    <Preset>slower</Preset>
                                </Video>
                                <Video>
                                    <Name>video_360</Name>
                                    <Codec>h264</Codec>
                                    <Height>360</Height>
                                    <Bitrate>700000</Bitrate>
                                    <Framerate>24</Framerate>
                                    <ThreadCount>4</ThreadCount>
                                    <Preset>slower</Preset>
                                </Video>

                                <Audio>
                                    <Name>bypass_audio</Name>
                                    <Bypass>true</Bypass>
                                </Audio>
                                <Audio>
                                    <Name>cond_audio_opus</Name>
                                    <Codec>opus</Codec>
                                    <Bitrate>128000</Bitrate>
                                    <Samplerate>48000</Samplerate>
                                    <Channel>2</Channel>
                                </Audio>
                            </Encodes>
                        </OutputProfile>
                    </OutputProfiles>

                    <Providers>
                        <RTMP>
                            <BlockDuplicateStreamName>true</BlockDuplicateStreamName>
                        </RTMP>
                    </Providers>

                    <Publishers>
                        <OVT />
                        <FILE>
                            <RootPath>/opt/ovenmediaengine/recordings</RootPath>
                            <FilePath>/${VirtualHost}/${Application}/${Stream}/manual-recording-${StartTime:YYYYMMDDhhmmss}.ts</FilePath>
                            <InfoPath>/${VirtualHost}/${Application}/${Stream}/manual-recording.xml</InfoPath>
                            <StreamMap>
                                <Enable>${env:ENABLE_AUTOMATED_RECORDING:false}</Enable>
                                <Path>./record.xml</Path>
                            </StreamMap>
                        </FILE>
                    </Publishers>
                </Application>
            </Applications>
        </VirtualHost>
    </VirtualHosts>
</Server>

I wasn't able to get any dumps from as the K8s container was deleted

dimiden commented 2 weeks ago

@bartvp1 Thank you for the configuration file. Could you also upload the logs?

bartvp1 commented 2 weeks ago

These are the startup logs, I wasn't able to retrieve any other logs before the crash since they havent been persisted

I [OvenMediaEngine:1] Config | config_manager.cpp:239  | Trying to set logfile in directory... (/var/log/ovenmediaengine)
W [OvenMediaEngine:1] Socket | socket.cpp:1092 | [#5] [0x7f79ada341d0] Could not send data: -1 ([errno] Network is unreachable (101))
W [OvenMediaEngine:1] WebRTC Publisher | webrtc_publisher.cpp:105  | WebRTC Publisher is disabled by configuration
W [OvenMediaEngine:1] LLHLS Publisher | llhls_publisher.cpp:78   | LLHLS Publisher is disabled by configuration
W [OvenMediaEngine:1] Thumbnail | thumbnail_publisher.cpp:79   | ThumbnailPublisher is disabled by configuration
W [OvenMediaEngine:1] HLS Publisher | hls_publisher.cpp:78   | HLS Publisher is disabled by configuration
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:54   | No supported NVIDIA Accelerator
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:64   | No supported Xilinx Media Accelerator
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:74   | No supported Intel QuickSync Accelerator
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:84   | No supported Netint VPU Accelerator
W [OvenMediaEngine:1] WebRTC Provider | webrtc_provider.cpp:139  | WebRTCProvider is disabled by configuration
W [OvenMediaEngine:1] MpegtsProvider | mpegts_provider.cpp:147  | MPEGTSProvider is disabled by configuration
W [OvenMediaEngine:1] SrtProvider | srt_provider.cpp:50   | SrtProvider is disabled by configuration
W [OvenMediaEngine:1] Publisher | publisher.cpp:68   | Push publisher is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91   | OVT provider is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91   | RTSP Pull provider is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91   | Scheduled provider is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91   | Multiplex provider is disabled in #default#app application, so it was not created
dimiden commented 2 weeks ago

@bartvp1 Thank you for your response. Since the code did not run as expected, we may need to pinpoint the error location through the crash dump or review the logs to identify the situation before the issue occurred.

If you could mount the logs and dump path to persistent storage and share the relevant details when the issue is reproduced, it would be greatly appreciated. (Even if the logs seem unnecessary, such as the OME version and host information, please do not remove them and upload the full logs.)

bartvp1 commented 2 weeks ago

Thank you, I'll post the logs here when this problem appears again!