search

ant-media / Ant-Media-Server

Ant Media Server is a live streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency. Ant Media Server is auto-scalable and it can run on-premise or on-cloud.
https://antmedia.io
Other
4.27k stars 627 forks source link

RTMP stream not showing on instances launched using AWS marketplace image (with & without GPU) #4028

Closed yashtandon113 closed 1 year ago

yashtandon113 commented 2 years ago

Short description

After launching the Ant media server using AWS marketplace image the RTMP stream is connected but not showing on dashboard but after restarting the server it started working and reflecting over dashboard.

Environment

Steps to reproduce

  1. Launch AMS through AWS marketplace image
  2. Publish RTMP stream

Expected behavior

Stream should show on dashboard panel.

Actual behavior

Stream is connected and publishing fine as well but not showing on dashboard but after restarting the server it works.

Logs

undefinedionContext - Refreshing Root WebApplicationContext: startup date [Sun Mar 06 19:13:18 UTC 2022]; parent: org.springframework.context.support.FileSystemXmlApplicationContext@3e906375 2022-03-06 19:13:18,264 [Loader:/LiveApp] INFO o.s.b.f.xml.XmlBeanDefinitionReader - Loading XML bean definitions from ServletContext resource [/WEB-INF/red5-web.xml] 2022-03-06 19:13:18,405 [Loader:/LiveApp] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring 2022-03-06 19:13:18,640 [Loader:/LiveApp] INFO org.red5.server.Context - Setting parent bean factory as core 2022-03-06 19:13:19,918 [Loader:/LiveApp] INFO i.a.datastore.db.DataStoreFactory - Used Datastore:mapdb db name:liveapp 2022-03-06 19:13:19,933 [Loader:/LiveApp] INFO org.red5.server.scope.WebScope - Set server [Server@4fa7daafmap['/root' -> 'default']] 2022-03-06 19:13:19,934 [Loader:/LiveApp] INFO org.red5.server.Server - Add mapping global: default host: context: LiveApp 2022-03-06 19:13:19,935 [Loader:/LiveApp] INFO i.a.AntMediaApplicationAdapter - Delete the ".closed" file in LiveApp 2022-03-06 19:13:19,940 [Loader:/LiveApp] INFO i.a.enterprise.webrtc.WebRTCAdaptor - setting excessive bandwidth algorithm enabled:false 2022-03-06 19:13:19,940 [Loader:/LiveApp] INFO i.a.AntMediaApplicationAdapter - LiveApp started 2022-03-06 19:13:19,948 [vert.x-eventloop-thread-1] INFO i.a.AntMediaApplicationAdapter - Stream source size: 0 2022-03-06 19:13:19,949 [vert.x-eventloop-thread-1] INFO i.a.s.StreamFetcherManager - StreamFetcherSchedule job name 6 2022-03-06 19:13:19,974 [Loader:/WebRTCAppEE] INFO o.s.w.c.s.XmlWebApplicationContext - Refreshing Root WebApplicationContext: startup date [Sun Mar 06 19:13:19 UTC 2022]; parent: org.springframework.context.support.FileSystemXmlApplicationContext@3e906375 2022-03-06 19:13:19,975 [Loader:/WebRTCAppEE] INFO o.s.b.f.xml.XmlBeanDefinitionReader - Loading XML bean definitions from ServletContext resource [/WEB-INF/red5-web.xml] 2022-03-06 19:13:20,128 [Loader:/WebRTCAppEE] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring 2022-03-06 19:13:20,202 [Loader:/WebRTCAppEE] INFO org.red5.server.Context - Setting parent bean factory as core 2022-03-06 19:13:20,445 [Loader:/WebRTCAppEE] INFO i.a.datastore.db.DataStoreFactory - Used Datastore:mapdb db name:webrtcappee 2022-03-06 19:13:20,450 [Loader:/WebRTCAppEE] INFO org.red5.server.scope.WebScope - Set server [Server@4fa7daafmap['/root' -> 'default', '/LiveApp' -> 'default']] 2022-03-06 19:13:20,450 [Loader:/WebRTCAppEE] INFO org.red5.server.Server - Add mapping global: default host: context: WebRTCAppEE 2022-03-06 19:13:20,451 [Loader:/WebRTCAppEE] INFO i.a.AntMediaApplicationAdapter - Delete the ".closed" file in WebRTCAppEE 2022-03-06 19:13:20,452 [Loader:/WebRTCAppEE] INFO i.a.enterprise.webrtc.WebRTCAdaptor - setting excessive bandwidth algorithm enabled:false 2022-03-06 19:13:20,452 [Loader:/WebRTCAppEE] INFO i.a.AntMediaApplicationAdapter - WebRTCAppEE started 2022-03-06 19:13:20,462 [vert.x-eventloop-thread-1] INFO i.a.AntMediaApplicationAdapter - Stream source size: 0 2022-03-06 19:13:20,462 [vert.x-eventloop-thread-1] INFO i.a.s.StreamFetcherManager - StreamFetcherSchedule job name 15 2022-03-06 19:14:12,259 [pool-4-thread-1] INFO org.red5.server.tomcat.WarDeployer - Deploy wars with app start 2022-03-06 19:18:09,844 [http-nio2-0.0.0.0-5080-exec-1] INFO io.antmedia.rest.RestServiceBase - Version Name 2.4.2.1 Version Type Enterprise Edition 2022-03-06 19:18:11,075 [http-nio2-0.0.0.0-5080-exec-10] INFO io.antmedia.statistic.GPUUtils - cuda initialized 2022-03-06 19:18:12,819 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - System cpu load:6 process cpu load:16 available memory: 15032716 KB used memory(RSS): 489372 KB 2022-03-06 19:18:12,820 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - Vertx worker queue size:0 WebRTCVertx worker queue size:0 2022-03-06 19:18:38,108 [http-nio2-0.0.0.0-5080-exec-6] WARN i.a.AntMediaApplicationAdapter - app settings bean updated for WebRTCAppEE 2022-03-06 19:19:00,673 [RTMPConnectionExecutor-1] INFO i.a.e.webrtc.WebRTCApplication - W3C x-category:session x-event:connect c-ip:103.87.56.50 c-client-id:0 2022-03-06 19:19:00,791 [RTMPConnectionExecutor-1] INFO i.a.s.AcceptOnlyStreamsWithWebhook - AcceptOnlyStreamsWithWebhook is not activated for stream testing webhook url: 2022-03-06 19:19:00,792 [RTMPConnectionExecutor-1] INFO i.a.s.AcceptOnlyStreamsInDataStore - AcceptOnlyStreamsInDataStore is not activated. Accepting all streams testing 2022-03-06 19:19:00,808 [RTMPConnectionExecutor-1] INFO org.red5.server.scope.BasicScope - ClientBroadcastStream already exists: org.red5.server.stream.ClientBroadcastStream@5d38f8fa new: org.red5.server.stream.ClientBroadcastStream@5d38f8fa 2022-03-06 19:19:00,809 [RTMPConnectionExecutor-1] INFO o.r.s.stream.ClientBroadcastStream - Stream start: testing 2022-03-06 19:19:00,813 [RTMPConnectionExecutor-1] INFO i.a.e.webrtc.WebRTCApplication - W3C x-category:stream x-event:publish c-ip:103.87.56.50 x-sname:8b73ad92-509a-4823-9d70-995885438f25 x-name:testing 2022-03-06 19:19:02,790 [DeadlockGuardScheduler-1] WARN o.r.s.n.r.ReceivedMessageTaskQueue - Interrupting unfinished active task [sessionId: ODAUQPMBSBQVV; packetNumber: 5; processing: true] 2022-03-06 19:19:02,800 [RTMPConnectionExecutor-2] INFO o.red5.server.stream.AbstractStream - metadata read data type -->>>> core map 2022-03-06 19:19:03,597 [RTMPConnectionExecutor-1] ERROR o.r.server.service.ServiceInvoker - Error executing call: Service: null Method: publish Num Params: 2 0: testing 1: live java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:189) at org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:213) at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:309) at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:115) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:72) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:38) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.UnsatisfiedLinkError: java.nio.channels.FileLockInterruptionException at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1739) at org.bytedeco.javacpp.Loader.load(Loader.java:1328) at org.bytedeco.javacpp.Loader.load(Loader.java:1132) at org.bytedeco.ffmpeg.global.avcodec.(avcodec.java:18) at io.antmedia.plugin.PacketFeeder.(PacketFeeder.java:27) at io.antmedia.muxer.MuxAdaptor.init(MuxAdaptor.java:354) at io.antmedia.enterprise.adaptive.EncoderAdaptor.init(EncoderAdaptor.java:183) at io.antmedia.muxer.MuxAdaptor.init(MuxAdaptor.java:308) at org.red5.server.stream.ClientBroadcastStream.startPublishing(ClientBroadcastStream.java:984) at org.red5.server.stream.StreamService.publish(StreamService.java:721) ... 16 common frames omitted Caused by: java.nio.channels.FileLockInterruptionException: null at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1138) at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1063) at org.bytedeco.javacpp.Loader.cacheResource(Loader.java:647) at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1582) ... 25 common frames omitted 2022-03-06 19:19:03,598 [RTMPConnectionExecutor-1] WARN o.r.s.net.rtmp.RTMPMinaConnection - Interrupted while waiting for write lock. State: connected java.lang.InterruptedException: null at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1367) at java.base/java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:415) at org.red5.server.net.rtmp.RTMPMinaConnection.write(RTMPMinaConnection.java:342) at org.red5.server.net.rtmp.Channel.write(Channel.java:139) at org.red5.server.net.rtmp.Channel.sendStatus(Channel.java:194) at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:314) at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:115) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:72) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:38) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2022-03-06 19:19:03,599 [RTMPConnectionExecutor-1] INFO o.r.s.net.rtmp.RTMPMinaConnection - Session id: ODAUQPMBSBQVV in queue size: 0 pending msgs: 1 last ping/pong: 0 2022-03-06 19:19:03,599 [RTMPConnectionExecutor-1] INFO o.r.s.net.rtmp.RTMPMinaConnection - Available permits - decoder: 1 encoder: 1

burak-58 commented 2 years ago

java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:189) at org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:213) at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:309) at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:115) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:72) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:38) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.UnsatisfiedLinkError: java.nio.channels.FileLockInterruptionException at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1739) at org.bytedeco.javacpp.Loader.load(Loader.java:1328) at org.bytedeco.javacpp.Loader.load(Loader.java:1132) at org.bytedeco.ffmpeg.global.avcodec.(avcodec.java:18) at io.antmedia.plugin.PacketFeeder.(PacketFeeder.java:27) at io.antmedia.muxer.MuxAdaptor.init(MuxAdaptor.java:354) at io.antmedia.enterprise.adaptive.EncoderAdaptor.init(EncoderAdaptor.java:183) at io.antmedia.muxer.MuxAdaptor.init(MuxAdaptor.java:308) at org.red5.server.stream.ClientBroadcastStream.startPublishing(ClientBroadcastStream.java:984) at org.red5.server.stream.StreamService.publish(StreamService.java:721) ... 16 common frames omitted Caused by: java.nio.channels.FileLockInterruptionException: null at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1138) at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1063) at org.bytedeco.javacpp.Loader.cacheResource(Loader.java:647) at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1582) ... 25 common frames omitted 2022-03-07 12:12:57,983 [RTMPConnectionExecutor-1] WARN o.r.s.net.rtmp.RTMPMinaConnection - Interrupted while waiting for write lock. State: connected java.lang.InterruptedException: null at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1367) at java.base/java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:415) at org.red5.server.net.rtmp.RTMPMinaConnection.write(RTMPMinaConnection.java:342) at org.red5.server.net.rtmp.Channel.write(Channel.java:139) at org.red5.server.net.rtmp.Channel.sendStatus(Channel.java:194) at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:314) at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:115) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:72) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:38) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

justinh24 commented 2 years ago

Probably related or the same as #3400

SelimEmre commented 2 years ago

It still continues on v2.4.3. @burak-58 FYI. ams.log

justinh24 commented 2 years ago

Any estimates on when this will be fixed? I am still running old versions because this bug prevents our automated system from working.

mekya commented 2 years ago

Hi @justinh24 ,

Do you have scenario that reproduces this problem? We don't have the scenario to reproduce it every time.

I think it will be easier for us if we can reproduce the problem.

kboek commented 1 year ago

It's possible to reproduce even with AMS 2.5.1. I use Hashicorp Packer to create the AMI from Ubuntu 20.04 LTS and in the Packer script I call a bash script to install AMS. When I afterwards launch an EC2 from the generated AMI, the issue does occur. Restarting the antmedia service on the box solves the problem.

yashtandon113 commented 1 year ago

FYI please

The solution is recommended to @kboek on the community post below. https://community.antmedia.io/topic/1549-invocationtargetexception-and-when-launching-from-custom-ami/#comment-5711

gorghino commented 7 months ago

@yashtandon113 Link is broken