search

robertdoo / red5

Automatically exported from code.google.com/p/red5
0 stars 0 forks source link

RTMPMinaConnection Incoming message handling failed #461

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Run my app (live RTMP-RTP streaming, similar to red5phone or BigBlueButton)
2. Setup a connection, which streams RTMP to RTP and RTP to RTMP
3. Wait a few minutes, connection is broken and exception is thrown

What is the expected output? What do you see instead?
Expected: connection running indefinetely. Instead it closes after an exception 
is thrown.

What version of the product are you using? On what operating system?
Red5 Server 1.0.2 $Rev: 4730 $

Please provide any additional information below.

Stack trace:

[NioProcessor-9] WARN  o.r.s.net.rtmp.RTMPMinaConnection Incoming message 
handling failed on G0IXZHNKTBZNF
    org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@34d4b411] did not accept task:
Task@7879e32f
    at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:244)
    at org.red5.server.net.rtmp.RTMPMinaConnection.handleMessageReceived(RTMPMinaConnection.java:151)
    at org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived(RTMPMinaIoHandler.java:162)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:690)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
    at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:407)
    at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:236)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
    at org.red5.server.net.rtmpe.RTMPEIoFilter.messageReceived(RTMPEIoFilter.java:132)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
    at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:410)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:710)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:664)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:653)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1124)
    at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:679)
    Caused by: java.util.concurrent.RejectedExecutionException: null
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1992)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:822)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1373)
    at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:241)
    at org.red5.server.net.rtmp.RTMPMinaConnection.handleMessageReceived(RTMPMinaConnection.java:151)
    at org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived(RTMPMinaIoHandler.java:162)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:690)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
    at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:407)
    at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:236)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
    at org.red5.server.net.rtmpe.RTMPEIoFilter.messageReceived(RTMPEIoFilter.java:132)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
    at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
    at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:410)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:710)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:664)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:653)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1124)
    at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:679)

Original issue reported on code.google.com by goz...@gmail.com on 6 Nov 2013 at 9:43

GoogleCodeExporter commented 9 years ago 
I tried messing with params in red5-core.xml as suggested in 
https://groups.google.com/forum/#!topic/red5interest/Yptgl1WnIN4.

I've tried:
 * upping queueCapacity to 16, 24 and 128
 * changing both corePoolSize and maxPoolSize to 2, 4, 8
 * changing only maxPoolSize to 8 (leaving corePoolSize at 1)

Nothing works (although leaving corePoolSize and maxPoolsize at 1 and upping 
queueCapacity to 128 seems to make the problem less frequent).
      <property name="corePoolSize" value="1" />
      <property name="maxPoolSize" value="1" />
      <property name="queueCapacity" value="128" />

Original comment by goz...@gmail.com on 6 Nov 2013 at 10:19

GoogleCodeExporter commented 9 years ago 
I suggest keeping the corePoolSize and maxPoolSize at "1"; if users want to 
make mods, it should only be to queueCapacity. I'm slightly puzzled by whats 
going on here since revision 4730 and later seem to have the best connection 
handling that I've seen.

Original comment by mondain on 11 Nov 2013 at 8:22

GoogleCodeExporter commented 9 years ago 
Can you reproduce the problem?

Original comment by goz...@gmail.com on 13 Nov 2013 at 10:28

GoogleCodeExporter commented 9 years ago 
I see the exception in my application, too, but I think this is a poor design 
in my application.

What happen is:
After the connection from my Flash client to my red5 application is 
established, the Flash client call invoke to my red5 application with same 
information.
Within the invokes method in my red5 application, I send a chat log with the 
invoke method back to the Flash client. The problem is now, if the chat log is 
full, I call the invoke method inside a loop 15 times and than add an entry in 
a database. The value from queueCapacity is 8. The Flash client answer now with 
15 result messages and in this situation the queue grow up to the queueCapacity 
and I see the exception.

Original comment by wer...@icm-world.com on 23 Nov 2013 at 9:16

GoogleCodeExporter commented 9 years ago 
I am also having this problem. Has this been solved?

Original comment by andrewbr...@gmail.com on 20 Jan 2014 at 7:17

GoogleCodeExporter commented 9 years ago 
i 've this issue too,using the newest source code(check out at 2014-02-23) , 
red5 server running in eclipse using embeded tomcat,
all the config using default value,never modified,
and the client is web flash and android air ,at the beginning ,it is ok, about 
several minites later, both have this exception
client code like this :

_h264Settings = new H264VideoStreamSettings();
_h264Settings.setProfileLevel(H264Profile.MAIN, H264Level.LEVEL_5_1);  
_h264Settings.setMode(320, 240, 15);  
_h264Settings.setQuality(0, 80);

_ns=new NetStream(_nc);
_ns.videoStreamSettings = _h264Settings;
_ns.addEventListener(NetStatusEvent.NET_STATUS , netStatus);
_ns.attachCamera(_cam); 
_ns.attachAudio(_mic);
_ns.publish("pcstream", "live");

Original comment by cankyk...@gmail.com on 23 Feb 2014 at 2:38

Attachments:

GoogleCodeExporter commented 9 years ago 
the file 'red520140223.log' is my last try's log , using chrome ,about 10 
minites,breaked...

Original comment by cankyk...@gmail.com on 23 Feb 2014 at 2:45

GoogleCodeExporter commented 9 years ago 
I got the same issue too under reversion 4730.
I've tried setting the properties in red5-core.xml as below
      <property name="corePoolSize" value="1" />
      <property name="maxPoolSize" value="1" />
      <property name="queueCapacity" value="128" />
but it did not change the problem too much.

I notice that the "Incoming message handling failed" information sometimes 
appeared while I published a audio netstream in a scope that contained two or 
more connections which published audio netstreams at the same time. And when 
the information appeared, the connections were disconnected.

In cases that I only published video netstreams, it worked fine.
However, once I attach audio to a published netstream, the warning may showed 
again and then disconnected.

This problem is totally a blocker for real-time interactive applications, hope 
it could be fixed soon.

By the way, I used web flash and android air as my clients.

Original comment by twinso...@gmail.com on 2 Apr 2014 at 6:23

GoogleCodeExporter commented 9 years ago 
This is a hard one to track down, I've never reproduced it locally that I can 
recall. I tried a test yesterday using 1.0.2-M2 and a queue capacity of 64 (the 
red5 server was hosted on AWS). I streamed for 3+ hours with 2 publishers and 3 
subscribers without a single exception.

Original comment by mondain on 2 Apr 2014 at 3:35

GoogleCodeExporter commented 9 years ago 
I found a way to reproduce this issue using bigbluebutton. You should not see 
this issue any longer starting with 1.0.2-M3.

Original comment by mondain on 4 Apr 2014 at 4:43

GoogleCodeExporter commented 9 years ago 
Hi, i'm new in the development cycle of Red5. HAa 1.0.2-M3 been released? i 
only can find 1.0.2-M2 and the correspondent SNAPSHOT for 1.0.3.

Thank you for your work!

Original comment by oriolm...@gmail.com on 27 May 2014 at 2:27

GoogleCodeExporter commented 9 years ago 
oriolmapu, please post questions on the list. There is no M3 for 1.0.2, for 
that version use RELEASE.

Original comment by mondain on 27 May 2014 at 3:29