Closed Hou5e closed 6 years ago
Same on Linux PC with 16GB RAM:
2018-08-11 05:32:54,323 INFO [main] n.core.a.n [DefaultBlockchain.java:556] Applying block...
2018-08-11 05:32:54,323 INFO [main] n.core.a.n [DefaultBlockchain.java:665] applyBlock called...
2018-08-11 05:32:54,324 INFO [main] n.core.a.n [DefaultBlockchain.java:725] Done checking PoP endorsements for expected endorsed block hashes...
2018-08-11 05:32:54,324 INFO [main] n.core.a.n [DefaultBlockchain.java:739] All bitcoin headers added? true
2018-08-11 05:32:54,324 INFO [main] n.core.a.n [DefaultBlockchain.java:758] Doing PoP calcs...
2018-08-11 05:32:54,333 INFO [main] n.core.a.n [DefaultBlockchain.java:773] Got pop outputs
2018-08-11 05:32:54,333 INFO [main] n.core.a.n [DefaultBlockchain.java:786] Done filtering outputs...
2018-08-11 05:32:54,333 INFO [main] n.core.a.n [DefaultBlockchain.java:791] Done playing outputs...
2018-08-11 05:32:54,333 INFO [main] n.core.a.n [DefaultBlockchain.java:835] Done with pop stuff...
2018-08-11 05:32:54,333 INFO [main] n.core.a.n [DefaultBlockchain.java:850] total PoW outputs calculated
2018-08-11 05:32:54,333 INFO [main] n.core.a.n [DefaultBlockchain.java:874] Validated all PoW outputs...
2018-08-11 05:32:54,335 INFO [main] n.core.a.n [DefaultBlockchain.java:894] Checked all PoP outputs...
2018-08-11 05:32:54,335 INFO [main] n.core.a.n [DefaultBlockchain.java:916] Applied all transactions...
2018-08-11 05:32:54,345 INFO [main] n.core.a.n [DefaultBlockchain.java:936] Put best known Bitcoin blocks...
2018-08-11 05:32:54,345 INFO [main] n.core.a.n [DefaultBlockchain.java:942] Adjusted ledger for all PoW outputs...
2018-08-11 05:32:54,345 INFO [main] n.core.a.n [DefaultBlockchain.java:946] Done applying block 00000000002F193468866CB2F3AF3145FEF3DBEA4DB0DB5E!
2018-08-11 05:32:54,345 INFO [main] n.core.a.n [DefaultBlockchain.java:558] Done applying block...
2018-08-11 05:32:54,346 INFO [main] n.core.a.n [DefaultBlockchain.java:1822] Added block directly... state: SUCCESSFULLY_ADDED_BEST_CHAIN
2018-08-11 05:32:54,346 INFO [main] n.core.a.n [DefaultBlockchain.java:1934] Added a new block (00000000002F193468866CB2F3AF3145FEF3DBEA4DB0DB5E @ 71900) to best chain.
2018-08-11 05:32:54,346 INFO [main] n.core.a.n [DefaultBlockchain.java:1939] Setting the local blockchain height...
2018-08-11 05:32:54,346 INFO [main] n.core.a.n [DefaultBlockchain.java:1953] Calculating block template for block 71901!
2018-08-11 05:32:54,346 INFO [main] n.core.a.n [DefaultBlockchain.java:380] Blockchain save started
----------------------------------------
ERROR Message in terminal window was here: java.lang.OutOfMemoryError: GC overhead limit exceeded
----------------------------------------
2018-08-11 05:33:38,555 INFO [pool-1-thread-1] n.Program [Program.java:54] ** Minute Marker: 41 **
2018-08-11 05:33:38,555 INFO [pool-1-thread-1] n.Program [Program.java:55] Free Memory: 4921898KB | MaxMemory: 7456768KB | TotalMemory: 5238272KB
2018-08-11 05:34:38,190 INFO [pool-1-thread-1] n.Program [Program.java:54] ** Minute Marker: 42 **
2018-08-11 05:34:38,190 INFO [pool-1-thread-1] n.Program [Program.java:55] Free Memory: 4921887KB | MaxMemory: 7456768KB | TotalMemory: 5238272KB
2018-08-11 05:35:38,190 INFO [pool-1-thread-1] n.Program [Program.java:54] ** Minute Marker: 43 **
2018-08-11 05:35:38,190 INFO [pool-1-thread-1] n.Program [Program.java:55] Free Memory: 4921874KB | MaxMemory: 7456768KB | TotalMemory: 5238272KB
Same on 3rd PC (Linux) now upgraded to have 16GB RAM:
06:17:41.741 [main] INFO nodecore.core.a.n - When adding block 71830 (000000000112515B7653F591A63D5A45833318422B7D8407), we are also adding 16 Bitcoin blocks.
06:17:41.741 [main] INFO nodecore.core.a.n - About to add block directly...
06:17:41.741 [main] INFO nodecore.core.a.n - Added block directly... state: DOES_NOT_FIT
06:17:42.378 [main] WARN nodecore.core.a.n - N102: DOES_NOT_FIT
06:17:42.378 [main] WARN nodecore.core.a.n - Block has been placed in the cache and another attempt will be made to re-add it
06:17:43.045 [main] INFO nodecore.core.a.n - Before adding block 71829(0000000000F1DC03D15B7FE8F9A87C32BE9CA0BF0E81343A), latest Bitcoin block known: 000000000000005665AE2F742264EB398BCB327E9112CC52F990BE59800ACE15
06:17:43.045 [main] INFO nodecore.core.a.n - About to process block cache...
06:17:43.046 [main] INFO nodecore.core.a.n - Done processing block cache...
06:17:43.046 [main] INFO nodecore.core.a.n - When adding block 71829 (0000000000F1DC03D15B7FE8F9A87C32BE9CA0BF0E81343A), we are also adding 15 Bitcoin blocks.
06:17:43.046 [main] INFO nodecore.core.a.n - About to add block directly...
06:17:43.046 [main] INFO nodecore.core.a.n - Added block directly... state: DOES_NOT_FIT
06:17:43.048 [main] WARN nodecore.core.a.n - N102: DOES_NOT_FIT
06:17:43.048 [main] WARN nodecore.core.a.n - Block has been placed in the cache and another attempt will be made to re-add it
06:17:43.723 [main] INFO nodecore.core.a.n - Before adding block 71828(000000000169D75ADBEC1686162A626FC6AD43D8BF6A1C0F), latest Bitcoin block known: 000000000000005665AE2F742264EB398BCB327E9112CC52F990BE59800ACE15
06:17:43.723 [main] INFO nodecore.core.a.n - About to process block cache...
06:17:43.723 [main] INFO nodecore.core.a.n - Done processing block cache...
06:17:43.723 [main] INFO nodecore.core.a.n - When adding block 71828 (000000000169D75ADBEC1686162A626FC6AD43D8BF6A1C0F), we are also adding 15 Bitcoin blocks.
06:17:43.723 [main] INFO nodecore.core.a.n - About to add block directly...
06:17:43.724 [main] INFO nodecore.core.a.n - Added block directly... state: DOES_NOT_FIT
06:17:44.405 [main] WARN nodecore.core.a.n - N102: DOES_NOT_FIT
06:17:44.405 [main] WARN nodecore.core.a.n - Block has been placed in the cache and another attempt will be made to re-add it
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at nodecore.core.transactions.DefaultTransaction.o(DefaultTransaction.java:658)
at nodecore.core.a.k.<init>(DefaultBlock.java:230)
at nodecore.core.a.k.<init>(DefaultBlock.java:421)
at nodecore.core.a.k.<init>(DefaultBlock.java:413)
at nodecore.core.a.k$$FastClassByGuice$$77582dfc.newInstance(<generated>)
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:776)
at com.sun.proxy.$Proxy16.a(Unknown Source)
at nodecore.core.a.n.h(DefaultBlockchain.java:193)
at nodecore.Program.a(Program.java:89)
at nodecore.Program.main(Program.java:161)
06:18:25.174 [pool-1-thread-1] INFO nodecore.Program - ** Minute Marker: 36 **
06:18:25.175 [pool-1-thread-1] INFO nodecore.Program - Free Memory: 5079818KB | MaxMemory: 7456768KB | TotalMemory: 5395968KB
06:19:24.866 [pool-1-thread-1] INFO nodecore.Program - ** Minute Marker: 37 **
06:19:24.866 [pool-1-thread-1] INFO nodecore.Program - Free Memory: 5079807KB | MaxMemory: 7456768KB | TotalMemory: 5395968KB
The second retry on the 3 PCs resulted the same way:
Windows PC:
07:00:58.421 [main] INFO nodecore.core.a.n - About to process block cache...
07:00:58.421 [main] INFO nodecore.core.a.n - Done processing block cache...
07:00:58.421 [main] INFO nodecore.core.a.n - When adding block 71304 (00000000018379FE8E5
5A0794980464F9BEB61E435D12677), we are also adding 9 Bitcoin blocks.
07:00:58.421 [main] INFO nodecore.core.a.n - About to add block directly...
07:00:58.421 [main] INFO nodecore.core.a.n - Added block directly... state: DOES_NOT_FIT
07:00:58.881 [main] WARN nodecore.core.a.n - N102: DOES_NOT_FIT
07:00:58.881 [main] WARN nodecore.core.a.n - Block has been placed in the cache and a
nother attempt will be made to re-add it
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)
at nodecore.core.transactions.DefaultTransaction.o(DefaultTransaction.java:658)
at nodecore.core.a.k.<init>(DefaultBlock.java:230)
at nodecore.core.a.k.<init>(DefaultBlock.java:421)
at nodecore.core.a.k.<init>(DefaultBlock.java:413)
at nodecore.core.a.k$$FastClassByGuice$$77582dfc.newInstance(<generated>)
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newIn
stance(DefaultConstructionProxyFactory.java:89)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.ja
va:111)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.ja
va:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindin
gImpl.java:268)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:
776)
at com.sun.proxy.$Proxy16.a(Unknown Source)
at nodecore.core.a.n.h(DefaultBlockchain.java:193)
at nodecore.Program.a(Program.java:89)
at nodecore.Program.main(Program.java:161)
07:01:21.839 [pool-1-thread-1] INFO nodecore.Program - ** Minute Marker: 34 **
07:01:21.839 [pool-1-thread-1] INFO nodecore.Program - Free Memory: 5249920KB | MaxMemory
: 7456768KB | TotalMemory: 5565440KB
07:02:21.500 [pool-1-thread-1] INFO nodecore.Program - ** Minute Marker: 35 **
07:02:21.500 [pool-1-thread-1] INFO nodecore.Program - Free Memory: 5249909KB | MaxMemory
: 7456768KB | TotalMemory: 5565440KB
Linux PC:
07:01:13.253 [main] INFO nodecore.core.a.n - About to process block cache...
07:01:13.253 [main] INFO nodecore.core.a.n - Done processing block cache...
07:01:13.253 [main] INFO nodecore.core.a.n - When adding block 71840 (0000000001837BC0AE03C1CFCD6FA4C1CC691A7EA733B2B7), we are also adding 10 Bitcoin blocks.
07:01:13.253 [main] INFO nodecore.core.a.n - About to add block directly...
07:01:13.253 [main] INFO nodecore.core.a.n - Added block directly... state: DOES_NOT_FIT
07:01:13.255 [main] WARN nodecore.core.a.n - N102: DOES_NOT_FIT
07:01:13.255 [main] WARN nodecore.core.a.n - Block has been placed in the cache and another attempt will be made to re-add it
07:01:14.573 [main] INFO nodecore.core.a.n - Before adding block 71840(00000000003EA9E24B26A81CCCDBBBE8517EF53BE3E7CED9), latest Bitcoin block known: 000000000000005665AE2F742264EB398BCB327E9112CC52F990BE59800ACE15
07:01:14.573 [main] INFO nodecore.core.a.n - About to process block cache...
07:01:14.573 [main] INFO nodecore.core.a.n - Done processing block cache...
07:01:14.573 [main] INFO nodecore.core.a.n - When adding block 71840 (00000000003EA9E24B26A81CCCDBBBE8517EF53BE3E7CED9), we are also adding 10 Bitcoin blocks.
07:01:14.574 [main] INFO nodecore.core.a.n - About to add block directly...
07:01:14.574 [main] INFO nodecore.core.a.n - Added block directly... state: DOES_NOT_FIT
07:01:14.576 [main] WARN nodecore.core.a.n - N102: DOES_NOT_FIT
07:01:14.576 [main] WARN nodecore.core.a.n - Block has been placed in the cache and another attempt will be made to re-add it
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.veriblock.core.utilities.Utility.bytesToHex(Utility.java:138)
at nodecore.core.transactions.DefaultTransaction.a(DefaultTransaction.java:474)
at nodecore.core.transactions.DefaultTransaction.a(DefaultTransaction.java:382)
at nodecore.core.transactions.DefaultTransaction.<init>(DefaultTransaction.java:327)
at nodecore.core.transactions.DefaultTransaction$$FastClassByGuice$$f0b59614.newInstance(<generated>)
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:776)
at com.sun.proxy.$Proxy17.a(Unknown Source)
at nodecore.core.a.k.b(DefaultBlock.java:429)
at nodecore.core.a.k$$Lambda$2/473073770.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at nodecore.core.a.k.<init>(DefaultBlock.java:430)
at nodecore.core.a.k.<init>(DefaultBlock.java:413)
at nodecore.core.a.k$$FastClassByGuice$$77582dfc.newInstance(<generated>)
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
07:02:00.253 [pool-1-thread-1] INFO nodecore.Program - ** Minute Marker: 37 **
07:02:00.253 [pool-1-thread-1] INFO nodecore.Program - Free Memory: 4909865KB | MaxMemory: 7456768KB | TotalMemory: 5225984KB
07:02:59.915 [pool-1-thread-1] INFO nodecore.Program - ** Minute Marker: 38 **
07:02:59.915 [pool-1-thread-1] INFO nodecore.Program - Free Memory: 4909855KB | MaxMemory: 7456768KB | TotalMemory: 5225984KB
Other Linux PC:
07:06:47.943 [main] INFO nodecore.core.a.n - About to process block cache...
07:06:47.943 [main] INFO nodecore.core.a.n - Done processing block cache...
07:06:47.943 [main] INFO nodecore.core.a.n - When adding block 71987 (00000000008D5700D6719B23ACE7C96E00EAF7C39D3B81C0), we are also adding 21 Bitcoin blocks.
07:06:47.943 [main] INFO nodecore.core.a.n - About to add block directly...
07:06:47.943 [main] INFO nodecore.core.a.n - Added block directly... state: DOES_NOT_FIT
07:06:48.627 [main] WARN nodecore.core.a.n - N102: DOES_NOT_FIT
07:06:48.627 [main] WARN nodecore.core.a.n - Block has been placed in the cache and another attempt will be made to re-add it
07:06:50.077 [main] INFO nodecore.core.a.n - Before adding block 71988(00000000000B12A400EB8811C97B6571A6D3B9DE7E298379), latest Bitcoin block known: 000000000000005A5C1AAE12EBDD0E165D5EF931E4D12879EF4EEC4CD66E9B51
07:06:50.077 [main] INFO nodecore.core.a.n - About to process block cache...
07:06:50.077 [main] INFO nodecore.core.a.n - Done processing block cache...
07:06:50.077 [main] INFO nodecore.core.a.n - When adding block 71988 (00000000000B12A400EB8811C97B6571A6D3B9DE7E298379), we are also adding 21 Bitcoin blocks.
07:06:50.077 [main] INFO nodecore.core.a.n - About to add block directly...
07:06:50.077 [main] INFO nodecore.core.a.n - Added block directly... state: DOES_NOT_FIT
07:06:50.778 [main] WARN nodecore.core.a.n - N102: DOES_NOT_FIT
07:06:50.778 [main] WARN nodecore.core.a.n - Block has been placed in the cache and another attempt will be made to re-add it
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.String.<init>(String.java:166)
at org.veriblock.core.utilities.Utility.bytesToHex(Utility.java:143)
at nodecore.core.i.e(VeriBlockPoPEndorsement.java:119)
at nodecore.core.a.v.<init>(PoPDatastore.java:69)
at nodecore.core.a.v.a(PoPDatastore.java:59)
at nodecore.core.a.k.<init>(DefaultBlock.java:252)
at nodecore.core.a.k.<init>(DefaultBlock.java:421)
at nodecore.core.a.k.<init>(DefaultBlock.java:413)
at nodecore.core.a.k$$FastClassByGuice$$77582dfc.newInstance(<generated>)
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:776)
at com.sun.proxy.$Proxy16.a(Unknown Source)
at nodecore.core.a.n.h(DefaultBlockchain.java:193)
at nodecore.Program.a(Program.java:89)
at nodecore.Program.main(Program.java:161)
07:07:16.336 [pool-1-thread-1] INFO nodecore.Program - ** Minute Marker: 41 **
07:07:16.337 [pool-1-thread-1] INFO nodecore.Program - Free Memory: 5068816KB | MaxMemory: 7456768KB | TotalMemory: 5385216KB
07:08:16.015 [pool-1-thread-1] INFO nodecore.Program - ** Minute Marker: 42 **
07:08:16.015 [pool-1-thread-1] INFO nodecore.Program - Free Memory: 5068802KB | MaxMemory: 7456768KB | TotalMemory: 5385216KB
Try to start the node from 0, remove the whole chain. In my case with linux and Java 8 and 10 y goes from 90% RAM to 6%.
The errors above were from using the default RAM usage setting: DEFAULT_JVM_OPTS="-Xmx8g" And, using Java 8.
It looks like most of these issues come from the default option of 8GB set in the 'nodecore.bat' (Windows), 'nodecore' (Linux) not being sufficient to reload the blockchain.
It is mentioned here: https://wiki.veriblock.org/index.php?title=HowTo_run_NodeCore#Java_out-of-memory But, the example doesn't give new users the answer of needing to change it to like: 16GB RAM PC: DEFAULT_JVM_OPTS="-Xmx15g" or for 32GB RAM PC: DEFAULT_JVM_OPTS="-Xmx25g"
It would be nice if the developers would change the default option to a value that will work. And, ideally work out a solution to not need that much RAM.
It sounds like running the latest Java 10 might help over running Java 8. I haven't tried it yet.
RAM usage in NodeCore has been significantly reduced since v0.2.16, and v0.3.0 generally has RAM requirements down below 2GB for a node.
This error message is only shown in the terminal window. Please add logging this error.
This still happens on a Windows 32GB of RAM machine when updating a partial blockchain from the previous version. (Also mentioned in Issue #95)
Error on screen only: OutOfMemoryError: GC overhead limit exceeded:
Logfile attached: nodecore-v0.2.10rc8-OutOfMemoryError_GC overhead limit exceeded.log.txt