search

noushadali / mediaserver

Automatically exported from code.google.com/p/mediaserver
0 stars 0 forks source link

Mediaserver stop responding #114

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Media server installed on an Amazon EC2 instance (m1.large) together with MSS, 
but same issue occurred when MMS was the only running jvm process on the 
instance (seperate server).

After few days uptime (last time issue happened server was running for 5 days) 
the media server stop responding to any requests. Even the server's heartbeat 
that is suppose to execute every 1 min, stops. Also notice that there was 
almost no traffic to the server (only 2 calls few days before the server 
stopped)

When in that condition, i got the jstack and jmap trace along with logs and OS 
statistics such as open ports (+udp receive queue), memory and cpu consuption. 

Inspecting the jstack trace, the WorkerThread threads are not waiting to 
acquire the lock that ConcurrentLinkedList currently has locked as it supposed 
to be so they don't wake up on notify. 
Checking the CriticalWorkingThread threads, they are blocked waiting for the 
object that ConcurrentLinkedList has acquired and this is the correct behavior 
as they will wake up on a notify when a new task arrive.

Find attached the jstack trace and i can send jmap, mms logs and OS logs.

Original issue reported on code.google.com by gvage...@gmail.com on 10 Jan 2013 at 10:06

Attachments:

GoogleCodeExporter commented 9 years ago

Original comment by oifa.yul...@gmail.com on 10 Jan 2013 at 10:10

GoogleCodeExporter commented 9 years ago 
Since objects queues waiting on disappears on EC2 , even while not released in 
any stage changed the queue to work based on Unsafe.park instead of object.wait.
Changed by revision d5e0b67bcb26.
Waiting for confirmation

Original comment by oifa.yul...@gmail.com on 12 Jan 2013 at 8:38

GoogleCodeExporter commented 9 years ago 
After completing testsuite for oob should put mms on hold for 2 weeks to 
confirm that issue is gone. Currently looks good , also perfomance was 
increased due to park unpark usage

Original comment by oifa.yul...@gmail.com on 21 Jan 2013 at 1:34

GoogleCodeExporter commented 9 years ago 
Checked today , mms still responding , will recheck on 27/02 , if everything ok 
issue will be closed

Original comment by web...@web-ukraine.info on 20 Feb 2013 at 4:07

GoogleCodeExporter commented 9 years ago 
Confirmed that working

Original comment by oifa.yul...@gmail.com on 6 Mar 2013 at 1:37