search

cardano-foundation / cf-ledger-sync

An open-source Java application designed to facilitate access to indexed Cardano blockchain data
Apache License 2.0
19 stars 2 forks source link

Won't exit on OOM exception #111

Closed rcmorano closed 5 months ago

rcmorano commented 7 months ago

Aloha team,

I've come across with this [0] exception and would have expected the main thread to, say, exit 1 but it's not and it's hanging after the shutdown message as you can see. This is inconvenient as, for example, while running within a container, the orchestrator won't automatically restart it as the process is somehow still alive.

[0]

2024-01-19 05:34:46.595  INFO 1 --- [TaskScheduler-2] c.b.c.y.s.c.s.CursorCleanupScheduler     : Deleted 76 cursors before block 1677215
2024-01-19 05:35:53.175 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener method
HandlerMethod details:
Bean [org.cardanofoundation.ledgersync.explorerconsumer.listeners.BlockEventListener$$SpringCGLIB$$0]
Method [public void org.cardanofoundation.ledgersync.explorerconsumer.listeners.BlockEventListener.handleRollback(com.bloxbean.cardano.yaci.store.events.RollbackEvent)]
Resolved arguments:
[0] [type=com.bloxbean.cardano.yaci.store.events.RollbackEvent] [value=RollbackEvent(rollbackTo=[slot=38985997, hash='05bd0981251b56fa8d5c8e87ab30f181102c5a3546e3056728d31d93e6756355'], currentPoint=[slot=38986451, hash='7124810680b0c2879526f0690dea7120d7c69b45ca422ba057c286ee98d55535'], currentBlock=1679397, remotePublish=false)]

    at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:365)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:233)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:174)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:167)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:437)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:383)
    at com.bloxbean.cardano.yaci.store.core.service.StartService.start(StartService.java:126)
    at com.bloxbean.cardano.yaci.store.core.service.AppcationEventListener.initialize(AppcationEventListener.java:23)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:348)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:233)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:174)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:167)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:437)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370)
    at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109)
    at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
    at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
    at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:334)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298)
    at org.cardanofoundation.ledgersync.explorerconsumer.ExplorerConsumerApplication.main(ExplorerConsumerApplication.java:28)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:95)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.base/java.math.BigInteger.valueOf(BigInteger.java:1203)
    at java.base/java.math.BigDecimal.inflated(BigDecimal.java:4521)
    at java.base/java.math.BigDecimal.toBigInteger(BigDecimal.java:3551)
    at org.jooq.impl.DefaultBinding$DefaultBigIntegerBinding.get0(DefaultBinding.java:1806)
    at org.jooq.impl.DefaultBinding$DefaultBigIntegerBinding.get0(DefaultBinding.java:1763)
    at org.jooq.impl.DefaultBinding$InternalBinding.get(DefaultBinding.java:1071)
    at org.jooq.impl.CursorImpl$CursorRecordInitialiser.setValue(CursorImpl.java:1581)
    at org.jooq.impl.CursorImpl$CursorRecordInitialiser.apply(CursorImpl.java:1517)
    at org.jooq.impl.CursorImpl$CursorRecordInitialiser.apply(CursorImpl.java:1432)
    at org.jooq.impl.RecordDelegate.operate(RecordDelegate.java:144)
    at org.jooq.impl.CursorImpl$CursorIterator.fetchNext(CursorImpl.java:1389)
    at org.jooq.impl.CursorImpl$CursorIterator.hasNext(CursorImpl.java:1365)
    at org.jooq.impl.CursorImpl.fetchNext(CursorImpl.java:173)
    at org.jooq.impl.AbstractCursor.fetch(AbstractCursor.java:177)
    at org.jooq.impl.AbstractCursor.fetch(AbstractCursor.java:88)
    at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:265)
    at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:357)
    at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:290)
    at org.jooq.impl.SelectImpl.fetch(SelectImpl.java:2838)
    at org.cardanofoundation.ledgersync.explorerconsumer.repository.custom.CustomAddressTokenBalanceRepository.findAllByAddressMultiAssetIdPairIn(CustomAddressTokenBalanceRepository.java:68)
    at java.base/java.lang.invoke.LambdaForm$DMH/0x00007f4cf409c400.invokeVirtual(LambdaForm$DMH)
    at java.base/java.lang.invoke.LambdaForm$MH/0x00007f4cf4095400.invoke(LambdaForm$MH)
    at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)

2024-01-19 05:35:53.196  INFO 1 --- [           main] c.b.c.yaci.core.network.NodeClient       : Shutdown connection !!!
2024-01-19 05:35:53.197  INFO 1 --- [           main] c.b.cardano.yaci.core.network.Session    : Disposing the session !!!
2024-01-19 05:35:53.197  INFO 1 --- [ntLoopGroup-3-1] c.b.c.yaci.core.network.NodeClient       : Connection closed !!!
2024-01-19 05:35:53.220  INFO 1 --- [           main] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2024-01-19 05:35:53.226  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : explorer-consumer-pool - Shutdown initiated...
2024-01-19 05:35:53.241  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : explorer-consumer-pool - Shutdown completed.
satran004 commented 7 months ago

We will implement health endpoint to get the current status https://github.com/cardano-foundation/cf-ledger-sync/issues/49

satran004 commented 5 months ago

Already fixed.