search

ant-media / Ant-Media-Server

Ant Media Server is a live streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency. Ant Media Server is auto-scalable and it can run on-premise or on-cloud.
https://antmedia.io
Other
4.28k stars 631 forks source link

DB error happening when disk size is not enough #2010

Open SelimEmre opened 4 years ago

SelimEmre commented 4 years ago

Describe the bug I am getting error that unable to create stream , after service restart working fine. Using Enterprise Edition 1.9.1

To Reproduce

  1. Adaptive Streaming with 240p, 360p, 480p and 720p default values and WebRTC Frame Rate: 30 Record Live Streams as MP4: enabled. Add Date-Time to MP4 File names: enabled. VoD Streaming folder: /home/tortoise/video

Expected behavior It seems errors occur when there is no disk space.

Additional context Logs were added by sorted date.

2020-03-30 20:39:36,550 [https-jsse-nio-0.0.0.0-5443-exec-7] ERROR o.a.c.c.C.[.[.0.0.0.[.[default] - Servlet.service() for servlet [default] in context with path [/WebRTCAppEE] threw exception java.lang.StringIndexOutOfBoundsException: String index out of range: -12 at java.lang.String.substring(String.java:1931) at io.antmedia.filter.TokenFilterManager.getStreamId(TokenFilterManager.java:133) at io.antmedia.filter.TokenFilterManager.doFilter(TokenFilterManager.java:46) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

2020-04-01 08:01:46,219 [Red5_Scheduler_Worker-29] ERROR o.r.s.s.QuartzSchedulingServiceJob - Job io.antmedia.statistic.HlsViewerStats$$Lambda$168/95869448@6ec8b014 execution failed org.mapdb.DBException$VolumeIOError: null at org.mapdb.volume.MappedFileVol.ensureAvailable(MappedFileVol.java:155) at org.mapdb.WriteAheadLog.startNextFile(WriteAheadLog.java:132) at org.mapdb.WriteAheadLog.ensureFileReady(WriteAheadLog.java:1009) at org.mapdb.WriteAheadLog.walPutRecord(WriteAheadLog.java:955) at org.mapdb.StoreWAL.updateProtected(StoreWAL.kt:489) at org.mapdb.StoreWAL.update(StoreWAL.kt:426) at org.mapdb.BTreeMap.removeOrReplace(BTreeMap.kt:559) at org.mapdb.BTreeMap.replace(BTreeMap.kt:881) at io.antmedia.datastore.db.MapDBStore.updateHLSViewerCountLocal(MapDBStore.java:852) at io.antmedia.datastore.db.DataStore.updateHLSViewerCount(DataStore.java:228) at io.antmedia.statistic.HlsViewerStats.lambda$setApplicationContext$0(HlsViewerStats.java:120) at org.red5.server.scheduling.QuartzSchedulingServiceJob.executeInternal(QuartzSchedulingServiceJob.java:89) at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) Caused by: java.io.IOException: No space left on device at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:525) at org.mapdb.volume.RandomAccessFileVol.clearRAF(RandomAccessFileVol.java:281) at org.mapdb.volume.MappedFileVol.ensureAvailable(MappedFileVol.java:138) ... 14 common frames omitted

2020-04-06 12:55:08,844 [Red5_Scheduler_Worker-7] ERROR o.r.s.s.QuartzSchedulingServiceJob - Job io.antmedia.statistic.HlsViewerStats$$Lambda$181/847145612@49595475 execution failed java.lang.IndexOutOfBoundsException: Index: 14, Size: 1 at java.util.ArrayList.rangeCheck(ArrayList.java:657) at java.util.ArrayList.get(ArrayList.java:433) at java.util.Collections$SynchronizedList.get(Collections.java:2419) at org.mapdb.WriteAheadLog.walGetRecord(WriteAheadLog.java:863) at org.mapdb.StoreWAL.get(StoreWAL.kt:522) at org.mapdb.BTreeMap.getNode(BTreeMap.kt:800) at org.mapdb.BTreeMap.getNonBinary(BTreeMap.kt:271) at org.mapdb.BTreeMap.get(BTreeMap.kt:235) at io.antmedia.datastore.db.MapDBStore.get(MapDBStore.java:156) at io.antmedia.datastore.db.MapDBStore.updateHLSViewerCountLocal(MapDBStore.java:847) at io.antmedia.datastore.db.DataStore.updateHLSViewerCount(DataStore.java:228) at io.antmedia.statistic.HlsViewerStats.lambda$setApplicationContext$0(HlsViewerStats.java:120) at org.red5.server.scheduling.QuartzSchedulingServiceJob.executeInternal(QuartzSchedulingServiceJob.java:89) at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

2020-04-06 12:55:11,780 [https-jsse-nio-0.0.0.0-5443-exec-4] ERROR o.a.c.c.C.[.[.0.0.0.[.[jersey-serlvet] - Servlet.service() for servlet [jersey-serlvet] in context with path [] threw exception [java.lang.IndexOutOfBoundsException: Index: 14, Size: 1] with root cause java.lang.IndexOutOfBoundsException: Index: 14, Size: 1 at java.util.ArrayList.rangeCheck(ArrayList.java:657) at java.util.ArrayList.get(ArrayList.java:433) at java.util.Collections$SynchronizedList.get(Collections.java:2419) at org.mapdb.WriteAheadLog.walGetRecord(WriteAheadLog.java:863) at org.mapdb.StoreWAL.get(StoreWAL.kt:522) at org.mapdb.BTreeMap.getNode(BTreeMap.kt:800) at org.mapdb.BTreeMap.access$getNode(BTreeMap.kt:72) at org.mapdb.BTreeMap$BTreeIterator.advanceFrom(BTreeMap.kt:1031) at org.mapdb.BTreeMap$BTreeIterator.<init>(BTreeMap.kt:1016) at org.mapdb.BTreeMap$valueIterator$1.<init>(BTreeMap.kt:1223) at org.mapdb.BTreeMap.valueIterator(BTreeMap.kt:1223) at org.mapdb.BTreeMap$values$1.iterator(BTreeMap.kt:999) at io.antmedia.datastore.db.MapDBStore.getActiveBroadcastCount(MapDBStore.java:308) at io.antmedia.console.AdminApplication.getAppLiveStreamCount(AdminApplication.java:304) at io.antmedia.console.rest.RestService.getSystemResourcesInfo(RestService.java:541) at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at io.antmedia.console.rest.AuthenticationFilter.doFilter(AuthenticationFilter.java:33) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:364) at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:170) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.red5.logging.LoggerContextFilter.doFilter(LoggerContextFilter.java:84) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

mekya commented 4 years ago

What can we do if there is no space in the disk?