search

kaltura / platform-install-packages

Official deployment packages to install the Kaltura platform on a server or cluster environments using native OS package managers
GNU Affero General Public License v3.0
521 stars 242 forks source link

kaltura-red5 loading but Java CPU usage at 100% #599

Closed daemonhunt closed 7 years ago

daemonhunt commented 7 years ago

Hi Team!

Latest version of Kaltura CE (12.20.0) installed to Centos 6.9 64-bit and finally got it working like a charm. Took days to wrangle.

Everything works apart from Red5. Have reinstalled kaltura-red5 several times, even tried Red5 installs from official red5 site. Have tried different Java versions from Open JDK 1.6 and 1.7 to Oracle Java 8.

The problem is that the Red5 service starts fine, PID created successfully, no crashing. But visitng my site at port 5080 in a browser and the page never loads, it just loads forever. As soon as Red5 is started, Java usage stays 90%-100% the entire time, throughout the day and night.

Weirdly, no matter how I manipulate logback.xml, no log files are being created at all so cannot debug and I have tried multiple debugging settings. A strange folder is created under the root labelled tomcat.8080, have no idea why, nothing ever goes into it and other Red5 versions do not do this.

I have exhausted so many options, many hours scanning forums and what-not and I'm stuck. I have an older instance of Kaltura (9.x) with Red5 and this runs fine. I even tried using this version of Red5 on my newer server to no avail (same issue). We requre Red5 for recording of webcams, it's pretty crucial.

Any other suggestions or assistance out there? Could this be related to my new Kaltura CE instance being fully SSL? Should I build Red5 from source?

daemonhunt commented 7 years ago

I managed to find where the log files were going: ~/log

Fixed a couple of issues and all looks green, however the Java process is still running at 100% and I cannot access the site via ports 5080 or 5443.

This is the red5 startup log:

2017-07-30 14:57:50,308 [main] INFO org.red5.server.Launcher - Red5 Server 1.0.7 (https://github.com/Red5) 2017-07-30 14:57:50,369 [main] INFO o.s.c.s.FileSystemXmlApplicationContext - Refreshing org.springframework.context.support.FileSystemXmlApplicationContext@3f49dace: startup date [Sun Jul 30 14:57:50 NZST 2017]; root of context hierarchy 2017-07-30 14:57:50,386 [main] INFO o.s.b.f.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [red5.xml] 2017-07-30 14:57:50,505 [main] INFO o.s.b.f.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [jee-container.xml] 2017-07-30 14:57:50,577 [main] INFO o.s.b.f.c.PropertyPlaceholderConfigurer - Loading properties file from class path resource [red5.properties] 2017-07-30 14:57:50,598 [main] INFO o.r.spring.Red5ApplicationContext - setApplicationContext: org.springframework.context.support.FileSystemXmlApplicationContext@3f49dace: startup date [Sun Jul 30 14:57:50 NZST 2017]; root of context hierarchy 2017-07-30 14:57:50,599 [main] INFO o.r.spring.Red5ApplicationContext - Red5ApplicationContext init 2017-07-30 14:57:50,600 [main] INFO o.r.spring.Red5ApplicationContext - Refreshing ApplicationContext 'red5.common': startup date [Sun Jul 30 14:57:50 NZST 2017]; parent: org.springframework.context.support.FileSystemXmlApplicationContext@3f49dace 2017-07-30 14:57:50,600 [main] INFO o.s.b.f.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [red5-common.xml] 2017-07-30 14:57:50,659 [main] INFO o.s.b.f.c.PropertyPlaceholderConfigurer - Loading properties file from class path resource [red5.properties] 2017-07-30 14:57:50,690 [main] INFO org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor 2017-07-30 14:57:50,705 [main] INFO o.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl 2017-07-30 14:57:50,706 [main] INFO org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.2.2 created. 2017-07-30 14:57:50,706 [main] INFO org.quartz.simpl.RAMJobStore - RAMJobStore initialized. 2017-07-30 14:57:50,707 [main] INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.2.2) 'Red5_Scheduler' with instanceId 'NON_CLUSTERED' Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally. NOT STARTED. Currently in standby mode. Number of jobs executed: 0 Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 32 threads. Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

2017-07-30 14:57:50,707 [main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'Red5_Scheduler' initialized from the specified file : '/usr/lib/red5/conf/quartz.properties' from the class resource path. 2017-07-30 14:57:50,707 [main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.2.2 2017-07-30 14:57:50,707 [main] INFO org.quartz.core.QuartzScheduler - Scheduler Red5Scheduler$_NON_CLUSTERED started. 2017-07-30 14:57:50,715 [main] INFO o.s.r.rmi.RmiRegistryFactoryBean - Creating new RMI registry 2017-07-30 14:57:50,842 [main] INFO o.s.j.s.ConnectorServerFactoryBean - JMX connector server started: javax.management.remote.rmi.RMIConnectorServer@52af26ee 2017-07-30 14:57:51,053 [main] INFO o.s.j.e.a.AnnotationMBeanExporter - Registering beans for JMX exposure on startup 2017-07-30 14:57:51,055 [main] INFO o.s.j.e.a.AnnotationMBeanExporter - Bean with name 'org.springframework.jmx.support.ConnectorServerFactoryBean#0' has been autodetected for JMX exposure 2017-07-30 14:57:51,056 [main] INFO o.s.j.e.a.AnnotationMBeanExporter - Bean with name 'schedulingService' has been autodetected for JMX exposure 2017-07-30 14:57:51,056 [main] INFO o.s.j.e.a.AnnotationMBeanExporter - Bean with name 'clientBroadcastStream' has been autodetected for JMX exposure 2017-07-30 14:57:51,154 [main] INFO o.s.j.e.a.AnnotationMBeanExporter - Located MBean 'schedulingService': registering with JMX server as MBean [org.red5.server:name=schedulingService,type=QuartzSchedulingService] 2017-07-30 14:57:51,155 [main] INFO o.s.j.e.a.AnnotationMBeanExporter - Located MBean 'org.springframework.jmx.support.ConnectorServerFactoryBean#0': registering with JMX server as MBean [org.red5.server:name=org.springframework.jmx.support.ConnectorServerFactoryBean#0,type=RMIConnectorServer] 2017-07-30 14:57:51,161 [main] INFO o.r.spring.Red5ApplicationContext - setApplicationContext: org.springframework.context.support.FileSystemXmlApplicationContext@3f49dace: startup date [Sun Jul 30 14:57:50 NZST 2017]; root of context hierarchy 2017-07-30 14:57:51,161 [main] INFO o.r.spring.Red5ApplicationContext - Red5ApplicationContext init 2017-07-30 14:57:51,161 [main] INFO o.r.spring.Red5ApplicationContext - Refreshing ApplicationContext 'red5.core': startup date [Sun Jul 30 14:57:51 NZST 2017]; parent: ApplicationContext 'red5.common' 2017-07-30 14:57:51,162 [main] INFO o.s.b.f.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [red5-core.xml] 2017-07-30 14:57:51,201 [main] INFO o.s.b.f.c.PropertyPlaceholderConfigurer - Loading properties file from class path resource [red5.properties] 2017-07-30 14:57:51,221 [main] INFO o.r.s.net.rtmp.RTMPMinaTransport - RTMP will be bound to [0.0.0.0:1935] 2017-07-30 14:57:51,222 [main] INFO o.r.s.net.rtmp.RTMPMinaTransport - RTMP Mina Transport Settings Acceptor style: default I/O threads: 16 TCP no-delay: true keep-alive: false 2017-07-30 14:57:51,237 [main] INFO o.r.s.net.rtmp.RTMPMinaTransport - Traffic class modification is disabled 2017-07-30 14:57:51,237 [main] INFO o.r.s.net.rtmp.RTMPMinaTransport - Send buffer size: 65536 recv buffer size: 65536 so linger: -1 traffic class: 0 2017-07-30 14:57:51,264 [main] INFO org.red5.server.ContextLoader - ContextLoader init 2017-07-30 14:57:51,265 [main] INFO org.red5.server.ContextLoader - Loading: default.context = ${red5.root}/webapps/red5-default.xml => /usr/lib/red5/webapps/red5-default.xml 2017-07-30 14:57:51,266 [main] INFO o.s.c.s.FileSystemXmlApplicationContext - Refreshing org.springframework.context.support.FileSystemXmlApplicationContext@38aa816f: startup date [Sun Jul 30 14:57:51 NZST 2017]; parent: ApplicationContext 'red5.common' 2017-07-30 14:57:51,267 [main] INFO o.s.b.f.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [file:/usr/lib/red5/webapps/red5-default.xml] 2017-07-30 14:57:51,324 [main] INFO org.red5.server.Context - Setting parent bean factory as core 2017-07-30 14:57:51,356 [main] INFO org.red5.server.tomcat.TomcatLoader - Loading Tomcat 2017-07-30 14:57:51,356 [main] INFO org.red5.server.tomcat.TomcatLoader - Server root: /usr/lib/red5 2017-07-30 14:57:51,357 [main] INFO org.red5.server.tomcat.TomcatLoader - Config root: /usr/lib/red5/conf 2017-07-30 14:57:51,368 [main] INFO org.red5.server.tomcat.TomcatLoader - Application root: /usr/lib/red5/webapps 2017-07-30 14:57:51,416 [main] INFO org.red5.server.tomcat.TomcatLoader - Starting Tomcat servlet engine 2017-07-30 14:57:51,519 [main] INFO o.a.c.core.AprLifecycleListener - Loaded APR based Apache Tomcat Native library 1.1.34 using APR version 1.3.9. 2017-07-30 14:57:51,519 [main] INFO o.a.c.core.AprLifecycleListener - APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. 2017-07-30 14:57:51,532 [main] INFO o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-0.0.0.0-5080"] 2017-07-30 14:57:51,537 [main] INFO o.a.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read 2017-07-30 14:57:51,539 [main] INFO o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-0.0.0.0-5443"] 2017-07-30 14:57:51,788 [main] INFO o.a.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read 2017-07-30 14:57:51,794 [main] INFO o.a.catalina.core.StandardService - Starting service Tomcat 2017-07-30 14:57:51,794 [main] INFO o.a.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/8.0.33 2017-07-30 14:57:51,878 [0.0.0.0-startStop-1] INFO o.a.catalina.startup.ContextConfig - No global web.xml found

daemonhunt commented 7 years ago

Absolutely crazy. Discovered the issue and it was so tiny I missed it for hours and hours. I imagine this may be a bug in almost every modern version of Red5. Within the init.d script 'Red5', a short line is missing (which I have bolded below):

case "$1" in start) echo -n $"Starting $PROG: " cd $RED5_HOME $DAEMON >/dev/null 2>/dev/null & RETVAL=$? if [ $RETVAL -eq 0 ]; then echo $! > $PIDFILE touch /var/lock/subsys/$PROG

Added this and it's working perfectly. Hope this helps someone else as this issue appears in all the latest versions of Red5 that I have tried, including the one shipping with Kaltura CE 12.2.x

Also strange because Red5 is set as a system path. Best of luck!

jessp01 commented 7 years ago

Hi @daemonhunt,

kaltura-red5 is a deprecated package and is not longer actively used. Please see this thread: https://forum.kaltura.org/t/error-in-installing-kaltura-red5/3892/8

Thanks,