search

etclabscore / core-geth

A highly configurable Go implementation of the Ethereum protocol.
https://etclabscore.github.io/core-geth
GNU Lesser General Public License v3.0
270 stars 149 forks source link

Addition: Add support for Hypra and EthashB3 #617

Open JABirchall opened 8 months ago

JABirchall commented 8 months ago

This pull request is draft for now as we go through the steps to add support for Hypra network.

Hypra is a custom Proof-of-Work chain based on Ethereum, but recently we have had difficulty to upgrade the chain with upstream ethereum features. We want to include these changes in core-geth to make ethashb3 available for use in other chains and allow for easier upgrade of the Hypra network.

https://www.hypra.network/blog/the-future-of-geth-and-hypra.html

Our end goal is to add support for Hypra network to core-geth under the --hypra flag, and maintain our own version where Hypra is the default network, pulling down upstream updates and pushing updates when hypra adopts a fork.

Our current geth code avalible here: https://github.com/Rethereum-blockchain/go-rethereum

Merge checklist

JABirchall commented 8 months ago

Currently Hypra supports EVM upto Shanghai. Hypra does not support EIP-1559 -- This is planned to fork off of the main geth client and switch to core-geth. Or cancun.

JABirchall commented 8 months ago

Update, I found where to define ethashb3 engine creation.

IMO, i felt the CreateConsensusEngine was very busy and took it on my self to refactor that part so each engine is created in its own function and doesnt need more parameters passed through to 1 function, it now switches from the chainspec. This makes it easier and cleaner to add new censensus engines in future.

Feel free to review it, i can revert the change if you decide its not appropriate. Ive kept the orignal CreateConsensusEngine for legacy reasons and some part of the codebase still use it.

@meowsbits

https://github.com/etclabscore/core-geth/pull/617/commits/6bea66a49307fc981e84992cc3d18f4711cbc4a5

JABirchall commented 7 months ago

Just confirmed ethashb3 is now working and useable in a chain configuration.

JABirchall commented 7 months ago

Ok, im 99% sure ive completed everything needed for Hypra to be compatible. However, upon testing. CoreGeth cant sync from a Geth node which all our current nodes run on.

If i copy the chaindata into the coregeth directory it starts with all the blocks and another coregeth node can sync from that. But they are not able to connect to any geth nodes to stay in sync.

Any ideas @meowsbits this would be needed for transition from geth to core-geth.

JABirchall commented 7 months ago

From some testing ive done

meowsbits commented 7 months ago

@JABirchall

CoreGeth cant sync from a Geth node which all our current nodes run on.

Might be a Fork ID problem? On quick glance I didn't see any fork id files changed here yet... If you use --log.verbosity 5 you might be able to see the error to get at what's causing the failed conn.

meowsbits commented 7 months ago

If i copy the chaindata into the coregeth directory it starts with all the blocks

Just FYI, you could also use geth import to copy the (geth exported) chain data, which will have an added benefit of being a full regression test.

JABirchall commented 7 months ago

@JABirchall

CoreGeth cant sync from a Geth node which all our current nodes run on.

Might be a Fork ID problem? On quick glance I didn't see any fork id files changed here yet... If you use --log.verbosity 5 you might be able to see the error to get at what's causing the failed conn.

Yes, it does look like its the Fork ID

DEBUG[02-12|17:30:49.862] Adding p2p peer                          peercount=1 id=0d9d7e47165acc0a conn=dyndial addr=77.100.75.201:30303   name=Geth/v1.13.1-stable-...
TRACE[02-12|17:30:49.862] Starting protocol eth/68                 id=0d9d7e47165acc0a conn=dyndial
TRACE[02-12|17:30:49.863] Starting protocol snap/1                 id=0d9d7e47165acc0a conn=dyndial
INFO [02-12|17:30:49.878] WebSocket enabled                        url=ws://127.0.0.1:8547
INFO [02-12|17:30:49.878] HTTP server started                      endpoint=127.0.0.1:8547 auth=true  prefix=     cors=localhost vhosts=localhost
DEBUG[02-12|17:30:49.880] Ethereum handshake failed                id=0d9d7e47165acc0a conn=dyndial err="fork ID rejected: local incompatible or needs update"
TRACE[02-12|17:30:49.880] Protocol eth/68 failed                   id=0d9d7e47165acc0a conn=dyndial err="fork ID rejected: local incompatible or needs update"
DEBUG[02-12|17:30:49.880] Message handling failed in `snap`        peer=0d9d7e47 err=EOF
DEBUG[02-12|17:30:49.880] Removing p2p peer                        peercount=0 id=0d9d7e47165acc0a duration=18.249ms req=false err="fork ID rejected: local incompatible or needs update"
TRACE[02-12|17:30:49.902] << PONG/v4                               id=c473ce5a2ab716c2 addr=54.36.108.60:30303    err=nil
TRACE[02-12|17:30:49.903] << PING/v4                               id=c473ce5a2ab716c2 addr=54.36.108.60:30303    err=nil
TRACE[02-12|17:30:49.903] >> PONG/v4                               id=c473ce5a2ab716c2 addr=54.36.108.60:30303    err=nil
TRACE[02-12|17:30:49.938] Accepted connection                      addr=95.216.230.113:39096
TRACE[02-12|17:30:49.940] Failed RLPx handshake                    addr=95.216.230.113:39096  conn=inbound err="ecies: invalid message"
DEBUG[02-12|17:30:49.941] Adding p2p peer                          peercount=1 id=c473ce5a2ab716c2 conn=dyndial addr=54.36.108.60:30303    name=Geth/v1.13.1-stable-...
TRACE[02-12|17:30:49.941] Starting protocol eth/68                 id=c473ce5a2ab716c2 conn=dyndial
TRACE[02-12|17:30:49.941] Starting protocol snap/1                 id=c473ce5a2ab716c2 conn=dyndial
DEBUG[02-12|17:30:49.976] Ethereum handshake failed                id=c473ce5a2ab716c2 conn=dyndial err="fork ID rejected: local incompatible or needs update"
TRACE[02-12|17:30:49.976] Protocol eth/68 failed                   id=c473ce5a2ab716c2 conn=dyndial err="fork ID rejected: local incompatible or needs update"
DEBUG[02-12|17:30:49.976] Message handling failed in `snap`        peer=c473ce5a err=EOF
DEBUG[02-12|17:30:49.976] Removing p2p peer                        peercount=0 id=c473ce5a2ab716c2 duration=35.434ms req=false err="fork ID rejected: local incompatible or needs update"

What do i need to do to solve this?

Just FYI, you could also use geth import to copy the (geth exported) chain data, which will have an added benefit of being a full regression test.

I know, Its just quick to copy the data over for initial tests.

JABirchall commented 7 months ago

Just tried an export import. It failed saying block 1 merkle is invalid

WARN [02-13|09:55:11.325] Rewinding chain to upgrade configuration err="mismatching incompatible fork value: GetEthashEIP649Transition in database (have block 1001, want block <nil>, rewindto block 1000)"
INFO [02-13|09:55:12.749] Loaded most recent local header          number=1000  hash=97a02a..2cac1f td=7,717,656,495     age=6mo3w4d
INFO [02-13|09:55:12.749] Loaded most recent local block           number=0     hash=0fb784..8abdf8 td=9,035,329         age=6mo4w1d
INFO [02-13|09:55:12.750] Loaded most recent local snap block      number=1000  hash=97a02a..2cac1f td=7,717,656,495     age=6mo3w4d
INFO [02-13|09:55:12.750] Importing blockchain                     file=chainexport
ERROR[02-13|09:55:13.448] Failed to delete ethashb3 cache file     epoch=0 epochLength=32000 target.epoch=0 file=J:\rethereum\chaindata2\geth\ethashb3\cache-R23-0-0000000000000000 err="remove J:\\rethereum\\chaindata2\\geth\\ethashb3\\cache-R23-0-0000000000000000: The process cannot access the file because it is being used by another process."
ERROR[02-13|09:55:13.480]
########## BAD BLOCK #########
Block: 1 (0x1f00fb0110389b88301ff59b21741e65cb897d340319d07810709561d52092a9)
Error: invalid merkle root (remote: 5b106d1dcb576ae76f10bbee485803d24f0cce4f7404633aef53e49beeb13b29 local: b7c87ee12d4949e042bf509405665d2c11d81e158f2f71ee1afe31f25b77a973) dberr: %!w(<nil>)
Platform: geth (devel) go1.21.1 amd64 windows
VCS: 4eef8b7b-20240212 (dirty)
Chain config: &coregeth.CoreGethChainConfig{NetworkID:0x97ec5, ChainID:622277, SupportedProtocolVersions:[]uint{0x44, 0x43, 0x42}, EIP2FBlock:0, EIP7FBlock:0, DAOForkBlock:0, EIP150Block:0, EIP155Block:0, EIP160FBlock:0, EIP161FBlock:0, EIP170FBlock:0, EIP100FBlock:1001, EIP140FBlock:1001, EIP198FBlock:1001, EIP211FBlock:1001, EIP212FBlock:1001, EIP213FBlock:1001, EIP214FBlock:1001, eip649FInferred:false, EIP649FBlock:<nil>, EIP658FBlock:1001, EIP145FBlock:5503, EIP1014FBlock:5503, EIP1052FBlock:5503, eip1234FInferred:false, EIP1234FBlock:<nil>, EIP1283FBlock:5503, PetersburgBlock:5507, EIP152FBlock:5519, EIP1108FBlock:5519, EIP1344FBlock:5519, EIP1884FBlock:5519, EIP2028FBlock:5519, EIP2200FBlock:5519, EIP2200DisableFBlock:<nil>, eip2384Inferred:false, EIP2384FBlock:5521, eip3554Inferred:false, EIP3554FBlock:<nil>, eip4345Inferred:false, EIP4345FBlock:<nil>, EIP1706FBlock:<nil>, EIP2537FBlock:<nil>, ECIP1010PauseBlock:<nil>, ECIP1010Length:<nil>, ECIP1017FBlock:<nil>, ECIP1017EraRounds:<nil>, ECIP1080FBlock:<nil>, ECIP1099FBlock:<nil>, ECBP1100FBlock:<nil>, ECBP1100DeactivateFBlock:<nil>, EIP2315FBlock:<nil>, EIP2565FBlock:5527, EIP2718FBlock:5527, EIP2929FBlock:5527, EIP3198FBlock:1600957, EIP4399FBlock:<nil>, EIP2930FBlock:5527, EIP1559FBlock:13524557, EIP3541FBlock:<nil>, EIP3529FBlock:13524557, EIP5133FBlock:<nil>, eip5133Inferred:false, EIP3651FTime:(*uint64)(nil), EIP3855FTime:(*uint64)(nil), EIP3860FTime:(*uint64)(nil), EIP4895FTime:(*uint64)(nil), EIP6049FTime:(*uint64)(nil), EIP3651FBlock:<nil>, EIP3855FBlock:1600957, EIP3860FBlock:1600957, EIP4895FBlock:<nil>, EIP6049FBlock:<nil>, EIP4844FTime:(*uint64)(nil), EIP1153FTime:(*uint64)(nil), EIP5656FTime:(*uint64)(nil), EIP6780FTime:(*uint64)(nil), MergeNetsplitVBlock:<nil>, DisposalBlock:<nil>, Ethash:(*ctypes.EthashConfig)(nil), EthashB3:(*ctypes.EthashB3Config)(0x7ff64f5ba660), Clique:(*ctypes.CliqueConfig)(nil), Lyra2:(*ctypes.Lyra2Config)(nil), IsDevMode:false, TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, TrustedCheckpoint:(*ctypes.TrustedCheckpoint)(0x7ff64d394260), TrustedCheckpointOracle:(*ctypes.CheckpointOracleConfig)(nil), DifficultyBombDelaySchedule:ctypes.Uint64BigMapEncodesHex(nil), BlockRewardSchedule:ctypes.Uint64BigMapEncodesHex{0x0:4000000000000000000, 0xce5e4d:3000000000000000000, 0x19f0ab1:2000000000000000000, 0x26d6a73:1000000000000000000}, RequireBlockHashes:map[uint64]common.Hash{0x26160:0x2a27bec023108c5f650cb0c9b7aaae7e7fdeefbeb3cd14a8390bb0957043aca2, 0x7d000:0xd547c896967174295c61ea18534d3c69337a6fa6e957909dc29e4bcba873592a, 0x1ca91b:0x1834091726bd0890539db04a22b673a6d9262d8b350dffaefdb10abc72e2113d}, Lyra2NonceTransitionBlock:<nil>, HIPVeldinFBlock:500009}
Receipts:
##############################

ERROR[02-13|09:55:13.482] Import error                             err="invalid block 1: invalid merkle root (remote: 5b106d1dcb576ae76f10bbee485803d24f0cce4f7404633aef53e49beeb13b29 local: b7c87ee12d4949e042bf509405665d2c11d81e158f2f71ee1afe31f25b77a973) dberr: %!w(<nil>)"
ERROR[02-13|09:55:13.482] Failed to journal state snapshot         err="snapshot [0x198bd05357dc0603587918ca5ff78364e37fdf23fe00592ee30ea61b67726c31] missing"
INFO [02-13|09:55:13.482] Blockchain stopped

I dont know why this is, the hash is correct, chainspec is correct (apart from eip649). So not sure what causing the merkle to be invalid

Interestingly, EIP649 was not present in the chainconvert, but is a mismatch.

meowsbits commented 7 months ago

[re: forkid] What do i need to do to solve this?

The tests are here and theyre pretty self-explanatory. https://github.com/etclabscore/core-geth/blob/master/core/forkid/forkid_test.go. You can use these together with a known ForkID value (if you have one already for Hypra) to figure out where the mismatch is coming from.

This might help as well with the Merkle root mismatch, since forkids are derived from the fork sequence. The merkle root is the state root value, which tells you that something is wrong with the state. The error being not a gas used mismatch error, I'd start by asserting the genesis and her state values are what you expect, then running the single block#1 import as an adhoc (or permanent) test to be able to use debug tools to track down the code path.

JABirchall commented 7 months ago

[re: forkid] What do i need to do to solve this?

The tests are here and theyre pretty self-explanatory. https://github.com/etclabscore/core-geth/blob/master/core/forkid/forkid_test.go. You can use these together with a known ForkID value (if you have one already for Hypra) to figure out where the mismatch is coming from.

This might help as well with the Merkle root mismatch, since forkids are derived from the fork sequence. The merkle root is the state root value, which tells you that something is wrong with the state. The error being not a gas used mismatch error, I'd start by asserting the genesis and her state values are what you expect, then running the single block#1 import as an adhoc (or permanent) test to be able to use debug tools to track down the code path.

Thanks, i didnt know that geth had this, so its not known what the Hypra fork_id is, but i assume it easy to get.

JABirchall commented 7 months ago

Ok, i know what the problem is, forkid only hashes forks with the suffix Time or Block in chainconfig. On Hypra we put our Veldin and Gaspar forks in a sub struct of config for our specific forks, making these not part of the forkid hash.

While Veldin was out own fork to modify uncle rewards, Gaspar fork was to enable Shangai EVM upgrade (Core-Geth EIP3855FBlock, EIP3860FBlock and EIP3198FBlock) which is whats contributing to the invalid forkid.

https://github.com/Rethereum-blockchain/go-rethereum/blob/master/params/config.go#L75-L78

JABirchall commented 7 months ago

I have added a function to the configurator incompatibleNetworkNameMap for networks such as Hypra that do custom forks which dont follow convention.

This will ommit getters from the forks list of the specified map for a specific ChainID.

JABirchall commented 7 months ago

We got status syncing! Hope it completes succesfully

Update: Sync failed. It is rejecting blocks in the last 0.1%. 

########## BAD BLOCK #########
Block: 2057822 (0xa390d7e594a22a36b71c2eb49d1440304b9fe624207e5e41ca38a448905a4761)
Error: invalid merkle root (remote: 855a1aa836f85b69d3e6b3a194a6f532a833ee50c6428f2e66649351d45f3c09 local: 55cbc7290c77de6fd2f0ad48a5572f8a1abdf7f09e8b37ed05a9f30d07034019) dberr: %!w(<nil>)
Platform: geth (devel) go1.21.1 amd64 windows
VCS: 3966827a-20240213 (dirty)
Chain config: &coregeth.CoreGethChainConfig{NetworkID:0x97ec5, ChainID:622277, SupportedProtocolVersions:[]uint{0x44, 0x43, 0x42}, EIP2FBlock:0, EIP7FBlock:0, DAOForkBlock:0, EIP150Block:0, EIP155Block:0, EIP160FBlock:0, EIP161FBlock:0, EIP170FBlock:0, EIP100FBlock:1001, E
IP140FBlock:1001, EIP198FBlock:1001, EIP211FBlock:1001, EIP212FBlock:1001, EIP213FBlock:1001, EIP214FBlock:1001, eip649FInferred:false, EIP649FBlock:1001, EIP658FBlock:1001, EIP145FBlock:5503, EIP1014FBlock:5503, EIP1052FBlock:5503, eip1234FInferred:false, EIP1234FBlock:<n
il>, EIP1283FBlock:5503, PetersburgBlock:5507, EIP152FBlock:5519, EIP1108FBlock:5519, EIP1344FBlock:5519, EIP1884FBlock:5519, EIP2028FBlock:5519, EIP2200FBlock:5519, EIP2200DisableFBlock:<nil>, eip2384Inferred:true, EIP2384FBlock:5521, eip3554Inferred:false, EIP3554FBlock:
13524557, eip4345Inferred:false, EIP4345FBlock:<nil>, EIP1706FBlock:<nil>, EIP2537FBlock:<nil>, ECIP1010PauseBlock:<nil>, ECIP1010Length:<nil>, ECIP1017FBlock:<nil>, ECIP1017EraRounds:<nil>, ECIP1080FBlock:<nil>, ECIP1099FBlock:<nil>, ECBP1100FBlock:<nil>, ECBP1100Deactiva
teFBlock:<nil>, EIP2315FBlock:<nil>, EIP2565FBlock:5527, EIP2718FBlock:5527, EIP2929FBlock:5527, EIP3198FBlock:1600957, EIP4399FBlock:<nil>, EIP2930FBlock:5527, EIP1559FBlock:13524557, EIP3541FBlock:13524557, EIP3529FBlock:13524557, EIP5133FBlock:<nil>, eip5133Inferred:fal
se, EIP3651FTime:(*uint64)(nil), EIP3855FTime:(*uint64)(nil), EIP3860FTime:(*uint64)(nil), EIP4895FTime:(*uint64)(nil), EIP6049FTime:(*uint64)(nil), EIP3651FBlock:<nil>, EIP3855FBlock:1600957, EIP3860FBlock:1600957, EIP4895FBlock:<nil>, EIP6049FBlock:<nil>, EIP4844FTime:(*
uint64)(nil), EIP1153FTime:(*uint64)(nil), EIP5656FTime:(*uint64)(nil), EIP6780FTime:(*uint64)(nil), MergeNetsplitVBlock:<nil>, DisposalBlock:<nil>, Ethash:(*ctypes.EthashConfig)(nil), EthashB3:(*ctypes.EthashB3Config)(0x7ff7941da680), Clique:(*ctypes.CliqueConfig)(nil), L
yra2:(*ctypes.Lyra2Config)(nil), IsDevMode:false, TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, TrustedCheckpoint:(*ctypes.TrustedCheckpoint)(0x7ff791fb4260), TrustedCheckpointOracle:(*ctypes.CheckpointOracleConfig)(nil), DifficultyBombDelaySchedule:c
types.Uint64BigMapEncodesHex{0x1591:9000000}, BlockRewardSchedule:ctypes.Uint64BigMapEncodesHex{0x0:4000000000000000000, 0xce5e4d:3000000000000000000, 0x19f0ab1:2000000000000000000, 0x26d6a73:1000000000000000000}, RequireBlockHashes:map[uint64]common.Hash{0x26160:0x2a27bec
023108c5f650cb0c9b7aaae7e7fdeefbeb3cd14a8390bb0957043aca2, 0x7d000:0xd547c896967174295c61ea18534d3c69337a6fa6e957909dc29e4bcba873592a, 0x1ca91b:0x1834091726bd0890539db04a22b673a6d9262d8b350dffaefdb10abc72e2113d}, Lyra2NonceTransitionBlock:<nil>, HIPVeldinFBlock:500009}    
Receipts:
##############################

ERROR[02-13|18:49:19.469] Failed to retrieve beacon bounds for bad block reporting err="beacon sync not yet started"
WARN [02-13|18:49:19.469] Rewinding blockchain to block            target=2,055,835
INFO [02-13|18:49:19.593] Generated state snapshot                 accounts=13609         slots=13395         storage=1.59MiB dangling=0 elapsed=223.336ms
WARN [02-13|18:49:20.402] Dropping unsynced node during sync       id=365c2460f2d647a6 conn=dyndial addr=3.10.19.215:30306 type=Nodecrawler/v1.13.10...
INFO [02-13|18:49:26.970] Looking for peers                        peercount=2 tried=158 static=0
INFO [02-13|18:49:37.264] Looking for peers                        peercount=1 tried=72  static=0
INFO [02-13|18:49:47.286] Looking for peers                        peercount=1 tried=129 static=0
INFO [02-13|18:49:57.366] Looking for peers                        peercount=2 tried=118 static=0
INFO [02-13|18:50:07.409] Looking for peers                        peercount=2 tried=169 static=0
INFO [02-13|18:50:08.517] Loaded most recent local header          number=2,055,835 hash=78ab0e..334f04 td=11,389,592,339,834,949,303 age=4h56m20s
INFO [02-13|18:50:08.517] Loaded most recent local block           number=0         hash=0fb784..8abdf8 td=9,035,329                  age=7mo45m8s
INFO [02-13|18:50:08.519] Loaded most recent local snap block      number=2,055,835 hash=78ab0e..334f04 td=11,389,592,339,834,949,303 age=4h56m20s
INFO [02-13|18:50:08.519] Loaded last snap-sync pivot marker       number=2,057,821
WARN [02-13|18:50:08.519] Rolled back chain segment                header=2057884->2055835 snap=2057821->2055835 block=2057821->0 reason="syncing canceled (requested)"
WARN [02-13|18:50:08.520] Synchronisation failed, dropping peer    peer=c473ce5a2ab716c26a6f8b4ddb587ea24714d1ba736231b54bb58cc3f8928ba5 err="retrieved hash chain is invalid: invalid merkle root (remote: 855a1aa836f85b69d3e6b3a194a6f532a833ee50c6428f2e66649351d45f3c09 loca
l: 55cbc7290c77de6fd2f0ad48a5572f8a1abdf7f09e8b37ed05a9f30d07034019) dberr: %!w(<nil>)"
INFO [02-13|18:50:17.555] Looking for peers                        peercount=1 tried=143 static=0
INFO [02-13|18:50:19.744] Disabling direct-ancient mode            origin=2,055,835 ancient=1,441,791

This is confuing, since it made it past our last fork at 1,600,000~ So i am unsure whats causing the bad blocks. Especially since they are within the last 100 blocks needed to complete the sync

If using --syncmode=full or import it still fails on block 1

ERROR[02-13|18:56:30.183]
########## BAD BLOCK #########
Block: 1 (0x1f00fb0110389b88301ff59b21741e65cb897d340319d07810709561d52092a9)
Error: invalid merkle root (remote: 5b106d1dcb576ae76f10bbee485803d24f0cce4f7404633aef53e49beeb13b29 local: b7c87ee12d4949e042bf509405665d2c11d81e158f2f71ee1afe31f25b77a973) dberr: %!w(<nil>)
Platform: geth (devel) go1.21.1 amd64 windows
VCS: 3966827a-20240213 (dirty)
Chain config: &coregeth.CoreGethChainConfig{NetworkID:0x97ec5, ChainID:622277, SupportedProtocolVersions:[]uint{0x44, 0x43, 0x42}, EIP2FBlock:0, EIP7FBlock:0, DAOForkBlock:0, EIP150Block:0, EIP155Block:0, EIP160FBlock:0, EIP161FBlock:0, EIP170FBlock:0, EIP100FBlock:1001, EIP140FBlock:1001, EIP198FBlock:1001, EIP211FBlock:1001, EIP212FBlock:1001, EIP213FBlock:1001, EIP214FBlock:1001, eip649FInferred:false, EIP649FBlock:1001, EIP658FBlock:1001, EIP145FBlock:5503, EIP1014FBlock:5503, EIP1052FBlock:5503, eip1234FInferred:false, EIP1234FBlock:<nil>, EIP1283FBlock:5503, PetersburgBlock:5507, EIP152FBlock:5519, EIP1108FBlock:5519, EIP1344FBlock:5519, EIP1884FBlock:5519, EIP2028FBlock:5519, EIP2200FBlock:5519, EIP2200DisableFBlock:<nil>, eip2384Inferred:true, EIP2384FBlock:5521, eip3554Inferred:false, EIP3554FBlock:13524557, eip4345Inferred:false, EIP4345FBlock:<nil>, EIP1706FBlock:<nil>, EIP2537FBlock:<nil>, ECIP1010PauseBlock:<nil>, ECIP1010Length:<nil>, ECIP1017FBlock:<nil>, ECIP1017EraRounds:<nil>, ECIP1080FBlock:<nil>, ECIP1099FBlock:<nil>, ECBP1100FBlock:<nil>, ECBP1100DeactivateFBlock:<nil>, EIP2315FBlock:<nil>, EIP2565FBlock:5527, EIP2718FBlock:5527, EIP2929FBlock:5527, EIP3198FBlock:1600957, EIP4399FBlock:<nil>, EIP2930FBlock:5527, EIP1559FBlock:13524557, EIP3541FBlock:13524557, EIP3529FBlock:13524557, EIP5133FBlock:<nil>, eip5133Inferred:false, EIP3651FTime:(*uint64)(nil), EIP3855FTime:(*uint64)(nil), EIP3860FTime:(*uint64)(nil), EIP4895FTime:(*uint64)(nil), EIP6049FTime:(*uint64)(nil), EIP3651FBlock:<nil>, EIP3855FBlock:1600957, EIP3860FBlock:1600957, EIP4895FBlock:<nil>, EIP6049FBlock:<nil>, EIP4844FTime:(*uint64)(nil), EIP1153FTime:(*uint64)(nil), EIP5656FTime:(*uint64)(nil), EIP6780FTime:(*uint64)(nil), MergeNetsplitVBlock:<nil>, DisposalBlock:<nil>, Ethash:(*ctypes.EthashConfig)(nil), EthashB3:(*ctypes.EthashB3Config)(0x7ff7f813a680), Clique:(*ctypes.CliqueConfig)(nil), Lyra2:(*ctypes.Lyra2Config)(nil), IsDevMode:false, TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, TrustedCheckpoint:(*ctypes.TrustedCheckpoint)(0x7ff7f5f14260), TrustedCheckpointOracle:(*ctypes.CheckpointOracleConfig)(nil), DifficultyBombDelaySchedule:ctypes.Uint64BigMapEncodesHex{0x1591:9000000}, BlockRewardSchedule:ctypes.Uint64BigMapEncodesHex{0x0:4000000000000000000, 0xce5e4d:3000000000000000000, 0x19f0ab1:2000000000000000000, 0x26d6a73:1000000000000000000}, RequireBlockHashes:map[uint64]common.Hash{0x26160:0x2a27bec023108c5f650cb0c9b7aaae7e7fdeefbeb3cd14a8390bb0957043aca2, 0x7d000:0xd547c896967174295c61ea18534d3c69337a6fa6e957909dc29e4bcba873592a, 0x1ca91b:0x1834091726bd0890539db04a22b673a6d9262d8b350dffaefdb10abc72e2113d}, Lyra2NonceTransitionBlock:<nil>, HIPVeldinFBlock:500009}
Receipts:
##############################
JABirchall commented 7 months ago

core-geth Hypra is now fully syncing with the network!

JABirchall commented 7 months ago

How can I build core-geth for linux and mac? My normal build script cant build coregeth saying Lyra2 is undefined.

Builds fine on Windows

GOROOT=C:\Program Files\Go #gosetup
GOPATH=C:\Users\owner\go #gosetup
"C:\Program Files\Go\bin\go.exe" build -trimpath "-ldflags=-s -w" -o C:\Users\owner\Documents\projects\core-geth\bin\go_build_release_linux . #gosetup
github.com/ethereum/evmc/v7/bindings/go/evmc: build constraints exclude all Go files in C:\Users\owner\go\pkg\mod\github.com\ethereum\evmc\v7@v7.5.0\bindings\go\evmc
# github.com/ethereum/go-ethereum/consensus/lyra2
..\..\consensus\lyra2\api.go:15:9: undefined: Lyra2
..\..\consensus\lyra2\sealer.go:207:16: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:50:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:56:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:77:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:139:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:157:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:213:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:276:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:334:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:334:14: too many errors
JABirchall commented 7 months ago

Had a good run for 1 hour. But seems somewhere a a gas cost is different.

ERROR[02-13|23:45:13.652] 
########## BAD BLOCK #########
Block: 2059835 (0x5d7372b0590e19ae733c72f2f17a1ef53a9c716b328c190a3fdcfb88e36fa652)
Error: invalid gas used (remote: 178816 local: 208973)
Platform: geth (devel) go1.21.1 amd64 windows
VCS: 5a26d5db-20240213 (dirty)
Chain config: &coregeth.CoreGethChainConfig{NetworkID:0x97ec5, ChainID:622277, SupportedProtocolVersions:[]uint{0x44, 0x43, 0x42}, EIP2FBlock:0, EIP7FBlock:0, DAOForkBlock:<nil>, EIP150Block:0, EIP155Block:0, EIP160FBlock:0, EIP161FBlock:0, EIP170FBlock:0, EIP100FBlock:1001,
 EIP140FBlock:1001, EIP198FBlock:1001, EIP211FBlock:1001, EIP212FBlock:1001, EIP213FBlock:1001, EIP214FBlock:1001, eip649FInferred:false, EIP649FBlock:<nil>, EIP658FBlock:1001, EIP145FBlock:5503, EIP1014FBlock:5503, EIP1052FBlock:5503, eip1234FInferred:false, EIP1234FBlock:<
nil>, EIP1283FBlock:5503, PetersburgBlock:5507, EIP152FBlock:5519, EIP1108FBlock:5519, EIP1344FBlock:5519, EIP1884FBlock:5519, EIP2028FBlock:5519, EIP2200FBlock:5519, EIP2200DisableFBlock:<nil>, eip2384Inferred:true, EIP2384FBlock:5521, eip3554Inferred:false, EIP3554FBlock:1
3524557, eip4345Inferred:false, EIP4345FBlock:<nil>, EIP1706FBlock:<nil>, EIP2537FBlock:<nil>, ECIP1010PauseBlock:<nil>, ECIP1010Length:<nil>, ECIP1017FBlock:<nil>, ECIP1017EraRounds:<nil>, ECIP1080FBlock:<nil>, ECIP1099FBlock:<nil>, ECBP1100FBlock:<nil>, ECBP1100DeactivateF
Block:<nil>, EIP2315FBlock:<nil>, EIP2565FBlock:5527, EIP2718FBlock:5527, EIP2929FBlock:5527, EIP3198FBlock:1600957, EIP4399FBlock:<nil>, EIP2930FBlock:5527, EIP1559FBlock:<nil>, EIP3541FBlock:13524557, EIP3529FBlock:1600957, EIP5133FBlock:<nil>, eip5133Inferred:false, EIP36
51FTime:(*uint64)(nil), EIP3855FTime:(*uint64)(nil), EIP3860FTime:(*uint64)(nil), EIP4895FTime:(*uint64)(nil), EIP6049FTime:(*uint64)(nil), EIP3651FBlock:<nil>, EIP3855FBlock:1600957, EIP3860FBlock:1600957, EIP4895FBlock:<nil>, EIP6049FBlock:<nil>, EIP4844FTime:(*uint64)(nil
), EIP1153FTime:(*uint64)(nil), EIP5656FTime:(*uint64)(nil), EIP6780FTime:(*uint64)(nil), MergeNetsplitVBlock:<nil>, DisposalBlock:<nil>, Ethash:(*ctypes.EthashConfig)(nil), EthashB3:(*ctypes.EthashB3Config)(0x7ff6d752a680), Clique:(*ctypes.CliqueConfig)(nil), Lyra2:(*ctypes
.Lyra2Config)(nil), IsDevMode:false, TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, TrustedCheckpoint:(*ctypes.TrustedCheckpoint)(0x7ff6d5304260), TrustedCheckpointOracle:(*ctypes.CheckpointOracleConfig)(nil), DifficultyBombDelaySchedule:ctypes.Uint64Big
MapEncodesHex{0x1591:9000000}, BlockRewardSchedule:ctypes.Uint64BigMapEncodesHex{0x0:4000000000000000000, 0xce5e4d:3000000000000000000, 0x19f0ab1:2000000000000000000, 0x26d6a73:1000000000000000000}, RequireBlockHashes:map[uint64]common.Hash{0x26160:0x2a27bec023108c5f650cb0c9
b7aaae7e7fdeefbeb3cd14a8390bb0957043aca2, 0x7d000:0xd547c896967174295c61ea18534d3c69337a6fa6e957909dc29e4bcba873592a, 0x1ca91b:0x1834091726bd0890539db04a22b673a6d9262d8b350dffaefdb10abc72e2113d}, Lyra2NonceTransitionBlock:<nil>, HIPVeldinFBlock:500009}
Receipts:
  0: cumulative: 208973 gas: 208973 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xb442721e1c3d4bd79fdacd396acec2581c44b8e3af71f085b9258e3073098790 logs: [0xc0014842c0 0xc001484370 0xc001484420 0xc0014844d0 0xc001484580 0xc0014846e0 0xc001484790 0xc0014
84840 0xc0014848f0] bloom: 00000000000000000000000081000000000000000000000000000000000000000000000000000000000100002000000000000000000000000000000000200000000000000000000000000008000000000400000000400000000000000000000000080004020000000000000000000800000000000100040000000010
000000000000000000000000000020000800080080000004000000080000000000000000020000000800004000000000000000000000000100000000000000000000000000000002000000400000000000000000000000000000001000000002000020000010000000000080000000000040000020000008000000000008000000000000 state:    
##############################

Cant figure out which EIP is responsable for the gas difference

meowsbits commented 7 months ago

This is confuing, since it made it past our last fork at 1,600,000~ So i am unsure whats causing the bad blocks. Especially since they are within the last 100 blocks needed to complete the sync

If using --syncmode=full or import it still fails on block 1

The --syncmode snap protocol will not validate all blocks; only a random fraction on the way up, and all of last few . That's why you're seeing inconsistent fails. I recommend testing block validation exclusively with --syncmode full and/or import.

How can I build core-geth for linux and mac? My normal build script cant build coregeth saying Lyra2 is undefined.

Check the .github/workflows/ for build patterns. A go clean -cache and ensuring you're at the latest Go version never hurts.

Cant figure out which EIP is responsable for the gas difference

This is a tough problem. Many EIPs are gas cost adjustments. In order to debug you might try comparing block/tx trace results from the working client vs. the development one. With the A/B traces you can compare the block/tx opcode steps and their resulting states. Once you figure out which opcode step is mismatched, you can then cross-reference the target opcode against its associated gas-relevant EIPs.

JABirchall commented 7 months ago

The --syncmode snap protocol will not validate all blocks; only a random fraction on the way up, and all of last few . That's why you're seeing inconsistent fails. I recommend testing block validation exclusively with --syncmode full and/or import.

Ultimately i figured out the problem by not syncing in mining block 1, and found that the block rewards were wrong because schedulereward would only return custom rewards for ethash, so i had to add ethashb3 to that and it worked.

This is a tough problem. Many EIPs are gas cost adjustments. In order to debug you might try comparing block/tx trace results from the working client vs. the development one. With the A/B traces you can compare the block/tx opcode steps and their resulting states. Once you figure out which opcode step is mismatched, you can then cross-reference the target opcode against its associated gas-relevant EIPs.

im doing a full sync right now to find what point in the chain gas difference happens and with core-geth it is rather slow. Its only syncing 100 blocks every 8 seconds. With standard geth it does 2048 blocks every 2 seconds.

JABirchall commented 7 months ago

Update: Sync is now 200k blocks off our gaspar fork without any issues so i think that narrows it down to something within the london/merge/shangai evm upgrades that were part of Gaspar

First block from Gaspar upgrade which i deployed a shangai test contract failed to sync

ERROR[02-16|13:47:25.794]
########## BAD BLOCK #########
Block: 1600994 (0xce5bbe1fb36a5b33e55591fa026b38d5e03a53c9e23686e8225b2b01e89ed645)
Error: invalid bloom (remote: 00000000000000000000000001000000000000000000000000800000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000020000000000000000000800000000800000000000000000000000400000000000040000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000  local: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)
Platform: geth (devel) go1.21.1 amd64 windows
VCS: 5a26d5db-20240213 (dirty)
Chain config: &coregeth.CoreGethChainConfig{NetworkID:0x97ec5, ChainID:622277, SupportedProtocolVersions:[]uint{0x44, 0x43, 0x42}, EIP2FBlock:0, EIP7FBlock:0, DAOForkBlock:0, EIP150Block:0, EIP155Block:0, EIP160FBlock:0, EIP161FBlock:0, EIP170FBlock:0, EIP100FBlock:1001, EIP140FBlock:1001, EIP198FBlock:1001, EIP211FBlock:1001, EIP212FBlock:1001, EIP213FBlock:1001, EIP214FBlock:1001, eip649FInferred:false, EIP649FBlock:<nil>, EIP658FBlock:1001, EIP145FBlock:5503, EIP1014FBlock:5503, EIP1052FBlock:5503, eip1234FInferred:false, EIP1234FBlock:<nil>, EIP1283FBlock:5503, PetersburgBlock:5507, EIP152FBlock:5519, EIP1108FBlock:5519, EIP1344FBlock:5519, EIP1884FBlock:5519, EIP2028FBlock:5519, EIP2200FBlock:5519, EIP2200DisableFBlock:<nil>, eip2384Inferred:true, EIP2384FBlock:5521, eip3554Inferred:false, EIP3554FBlock:13524557, eip4345Inferred:false, EIP4345FBlock:<nil>, EIP1706FBlock:<nil>, EIP2537FBlock:<nil>, ECIP1010PauseBlock:<nil>, ECIP1010Length:<nil>, ECIP1017FBlock:<nil>, ECIP1017EraRounds:<nil>, ECIP1080FBlock:<nil>, ECIP1099FBlock:<nil>, ECBP1100FBlock:<nil>, ECBP1100DeactivateFBlock:<nil>, EIP2315FBlock:<nil>, EIP2565FBlock:5527, EIP2718FBlock:5527, EIP2929FBlock:5527, EIP3198FBlock:1600957, EIP4399FBlock:<nil>, EIP2930FBlock:5527, EIP1559FBlock:<nil>, EIP3541FBlock:13524557, EIP3529FBlock:<nil>, EIP5133FBlock:<nil>, eip5133Inferred:false, EIP3651FTime:(*uint64)(nil), EIP3855FTime:(*uint64)(nil), EIP3860FTime:(*uint64)(nil), EIP4895FTime:(*uint64)(nil), EIP6049FTime:(*uint64)(nil), EIP3651FBlock:<nil>, EIP3855FBlock:1600957, EIP3860FBlock:1600957, EIP4895FBlock:<nil>, EIP6049FBlock:<nil>, EIP4844FTime:(*uint64)(nil), EIP1153FTime:(*uint64)(nil), EIP5656FTime:(*uint64)(nil), EIP6780FTime:(*uint64)(nil), MergeNetsplitVBlock:<nil>, DisposalBlock:<nil>, Ethash:(*ctypes.EthashConfig)(nil), EthashB3:(*ctypes.EthashB3Config)(0x7ff75202b680), Clique:(*ctypes.CliqueConfig)(nil), Lyra2:(*ctypes.Lyra2Config)(nil), IsDevMode:false, TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, TrustedCheckpoint:(*ctypes.TrustedCheckpoint)(0x7ff74fe14220), TrustedCheckpointOracle:(*ctypes.CheckpointOracleConfig)(nil), DifficultyBombDelaySchedule:ctypes.Uint64BigMapEncodesHex{0x1591:9000000}, BlockRewardSchedule:ctypes.Uint64BigMapEncodesHex{0x0:4000000000000000000, 0xce5e4d:3000000000000000000, 0x19f0ab1:2000000000000000000, 0x26d6a73:1000000000000000000}, RequireBlockHashes:map[uint64]common.Hash{0x26160:0x2a27bec023108c5f650cb0c9b7aaae7e7fdeefbeb3cd14a8390bb0957043aca2, 0x7d000:0xd547c896967174295c61ea18534d3c69337a6fa6e957909dc29e4bcba873592a, 0x1ca91b:0x1834091726bd0890539db04a22b673a6d9262d8b350dffaefdb10abc72e2113d}, Lyra2NonceTransitionBlock:<nil>, HIPVeldinFBlock:500009}
Receipts:
  0: cumulative: 1326462 gas: 1326462 contract: 0x0376fC3D62F7a6C4e0707dE9aAb9988a708Be183 status: 0 tx: 0x15d76766b05cbd2ce01b23fa6469c412ee0ce093d8594081e80a44d0cec7ab7d logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
##############################

ERROR[02-16|13:47:25.796] Failed to retrieve beacon bounds for bad block reporting err="beacon sync not yet started"

https://explorer.hypra.network/block/1600994/transactions

JABirchall commented 7 months ago

Ok, i cant figure out what is or isnt enabled for the Gaspar fork to work.

Gaspar was just to upgrade the EVM on Hypra from Berlin to Shanghai, which as far as i can see is EIP3855, EIP3860, EIP3198, EIP3529.

JABirchall commented 7 months ago

The issue might be the IntrinsicGas function, as in our geth this is not enabled as part of enabling the EIP3860 EVM upgrade to CREATE and CREATE2 gas

https://github.com/etclabscore/core-geth/blob/master/core/state_transition.go#L103-L109

Looks like this EIP effects more than was activated on Hypra network.

meowsbits commented 7 months ago

Error: invalid bloom

... to me this suggests its not the gas (since that was ==), but the receipts or their transactions..

JABirchall commented 7 months ago

Error: invalid bloom

... to me this suggests its not the gas (since that was ==), but the receipts or their transactions..

* https://github.com/etclabscore/core-geth/blob/master/core/block_validator.go#L113-L123

* https://eips.ethereum.org/EIPS/eip-658 ?

Would the bloom be different because the transcation failed? As that transcation used 100% gas which succeeds on mainnet, but if the there is a gas difference. between whats expected and what core-geth gives, would give different bloom since the gas limit was reached so it cant report gas difference?

Another indicator of this is that the local reported state has empty logs, but that transcation should have 1 log [0x8be0079c] OwnershipTransferred(), https://explorer.hypra.network/tx/0x15d76766b05cbd2ce01b23fa6469c412ee0ce093d8594081e80a44d0cec7ab7d/logs

Do you have discord so we can have a call about it? Because im stuck, i have ideas, but testing some of them didnt change anything or completely invalidated the chain. I think doing a pair code to just get this over the line and syncing past Gaspar fork would be useful.

meowsbits commented 7 months ago

The issue might be the IntrinsicGas function, as in our geth this is not enabled as part of enabling the EIP3860 EVM upgrade to CREATE and CREATE2 gas [...] Looks like this EIP effects more than was activated on Hypra network.

Can you say more about this? I went to find your source here but they look identical to me.

Have you tried comparing results of trace_call on both clients?

I'm willing to meet, but would like a little more context about what you've tried and haven't yet.

JABirchall commented 7 months ago

Can you say more about this? I went to find your source here but they look identical to me.

That paramter is bound to Shanghai but we didnt activate shanghai we only activate the shanhai EVM code with our gaspar fork here and here. I missed that when doing Gaspar.

Have you tried comparing results of trace_call on both clients?

I havnt yet, not really sure how to since it rejects the block.

I'm willing to meet, but would like a little more context about what you've tried and haven't yet.

What ive tested is commited. The last thing im going to test is just defining gaspar in params and adding that to enable the evm opcodes. if thaty doesnt work im completely lost.

JABirchall commented 7 months ago

Ok, I fixed the EVM issue it looks like its now fully syncing.

Currently doing a full sync from start, will see if it keeps insync.

JABirchall commented 7 months ago

Ok, its still syncing, but very slowly.

There seems to be a performance issue with pebble, switching to leveldb and syncing is fast again.

Full import completed with no invalid blocks

✅ Status Syncs mainnet

Just need to figure out why its trying to generate the same dag multiple times and delete it multiple time. It was really bad on epoch 5