search

radixdlt / olympia-node

Radix monorepo
Other
142 stars 35 forks source link

Validator restarts #275

Open feeleep75 opened 3 years ago

feeleep75 commented 3 years ago

Hi - I am getting this kind of issues recently:

# 2021-05-12T03:56:21,606 [DEBUG/BFTSync/ConsensusRunner vb1q26cz] (BFTSync.java:443) - SYNC_VERTICES: Received GetVerticesErrorResponse: GetVerticesErrorResponse{vb1q0x47->HighQC[highest=QC{37:53223}, highestCommitted=<same>, highestTC=null]} highQC: HighQC[highest=QC{37:53223}, highestCommitted=<same>, highestTC=null]
java.lang.NullPointerException
    at com.radixdlt.consensus.sync.BFTSync.processGetVerticesLocalTimeout(BFTSync.java:318)
    at com.radixdlt.consensus.epoch.EpochManager.processGetVerticesLocalTimeout(EpochManager.java:408)
    at io.reactivex.rxjava3.internal.observers.LambdaObserver.onNext(LambdaObserver.java:63)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.drainNormal(ObservableObserveOn.java:201)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.run(ObservableObserveOn.java:255)
    at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:322)
    at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.runEager(ExecutorScheduler.java:287)
    at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:248)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
# 2021-05-12T03:56:21,607 [ERROR/EpochManagerRunner/ConsensusRunner vb1q26cz] (EpochManagerRunner.java:213) - Unexpected exception occurred
java.lang.NullPointerException: null
    at com.radixdlt.consensus.sync.BFTSync.processGetVerticesLocalTimeout(BFTSync.java:318) ~[radixdlt-1.0-beta.32.jar:?]
    at com.radixdlt.consensus.epoch.EpochManager.processGetVerticesLocalTimeout(EpochManager.java:408) ~[radixdlt-1.0-beta.32.jar:?]
    at io.reactivex.rxjava3.internal.observers.LambdaObserver.onNext(LambdaObserver.java:63) ~[rxjava-3.0.9.jar:?]
    at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.drainNormal(ObservableObserveOn.java:201) ~[rxjava-3.0.9.jar:?]
    at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.run(ObservableObserveOn.java:255) ~[rxjava-3.0.9.jar:?]
    at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:322) ~[rxjava-3.0.9.jar:?]
    at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.runEager(ExecutorScheduler.java:287) ~[rxjava-3.0.9.jar:?]
    at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:248) ~[rxjava-3.0.9.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
+ find /usr -type f -name java -exec setfattr -n user.pax.flags -v em {} ;
+ env
+ sort
CORE_GOSSIP_PORT=30000
HOME=/root
HOSTNAME=853478c1ba60
JAVA_OPTS=-server -Xms4g -Xmx4g -XX:+HeapDumpOnOutOfMemoryError -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts -Djavax.net.ssl.trustStoreType=jks -Djava.security.egd=file:/dev/urandom -Dcom.sun.management.jmxremote.port=9010 -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=core -agentlib:jdwp=transport=dt_socket,address=50505,suspend=n,server=y
# 2021-05-12T03:56:24,541 [INFO/MemoryLeakDetector/main] (MemoryLeakDetector.java:54) - Starting memory leak detector...
2021-05-12 03:56:24,548 Log4j2-TF-3-AsyncLoggerConfig-2 ERROR An exception occurred processing Appender FILE java.lang.NullPointerException
    at org.apache.logging.log4j.util.StackLocator.calcLocation(StackLocator.java:79)
    at org.apache.logging.log4j.util.StackLocatorUtil.calcLocation(StackLocatorUtil.java:78)
    at org.apache.logging.log4j.core.impl.Log4jLogEvent.getSource(Log4jLogEvent.java:693)
    at org.apache.logging.log4j.core.pattern.FileLocationPatternConverter.format(FileLocationPatternConverter.java:57)
    at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
    at org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:345)
    at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:244)
    at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:229)
    at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:197)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:181)
    at org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender.append(RollingRandomAccessFileAppender.java:252)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders(AsyncLoggerConfig.java:127)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(AsyncLoggerConfig.java:121)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread(AsyncLoggerConfig.java:169)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:111)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:97)
    at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
    at java.base/java.lang.Thread.run(Thread.java:834)

2021-05-12 03:56:24,554 Log4j2-TF-3-AsyncLoggerConfig-2 ERROR An exception occurred processing Appender STDOUT java.lang.NullPointerException
    at org.apache.logging.log4j.util.StackLocator.calcLocation(StackLocator.java:79)
    at org.apache.logging.log4j.util.StackLocatorUtil.calcLocation(StackLocatorUtil.java:78)
    at org.apache.logging.log4j.core.impl.Log4jLogEvent.getSource(Log4jLogEvent.java:693)
    at org.apache.logging.log4j.core.pattern.FileLocationPatternConverter.format(FileLocationPatternConverter.java:57)
    at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
    at org.apache.logging.log4j.core.pattern.HighlightConverter.format(HighlightConverter.java:245)
    at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
    at org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:345)
    at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:244)
    at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:229)
    at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:197)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:181)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders(AsyncLoggerConfig.java:127)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(AsyncLoggerConfig.java:121)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread(AsyncLoggerConfig.java:169)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:111)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:97)
    at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
    at java.base/java.lang.Thread.run(Thread.java:834)
# 2021-05-12T03:56:24,641 [INFO/PersistedProperties/main] (PersistedProperties.java:61) - Loaded properties from default.config
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/radixdlt/lib/guice-4.2.3.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
cknowles commented 2 years ago

Was getting the same error for the log4j file appender part in the current 1.0.0 Docker setup. The node still appears to carry on and works, however I ended up customising the log4j configuration to get rid of it. Turned off the file appender at the same time since I didn't want it writing a file inside the container (not good practice).