search

ava-labs / subnet-evm

Launch your own EVM as an Avalanche Subnet
https://docs.avax.network/subnets/create-a-fuji-subnet
GNU Lesser General Public License v3.0
244 stars 223 forks source link

Invalid bloom error thus shutting down chain bootstrap #1114

Closed thesoulkiller closed 6 months ago

thesoulkiller commented 8 months ago

Short story: We recently missed the the major Warp messaging (Fuji) update, the subnet stopped 3 days, after we synchronized all of the archieve nodes and rpc's with copying the db. However now a new rpc nodes which tries to enter the subnet receives Fatal error and shuts down the bootstrapping process.

Describe the bug Somehow subnet produces a dirty block and inserts through all of the validators, then newly entered node fails to bootstrap the subnet chain. I am sharing the both of the archive validator node and a recent node ( which is just indexing node ) logs. After creating a certain block ( 2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf ) the validator kills the subnet subprocess.

The last line of the recently joined rpc node gives the following error and shuts down

FATAL <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> handler/handler.go:337 shutting down chain {"reason": "received an unexpected error", "error": "failed to execute job 2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf due to failed to verify block in bootstrapping: rpc error: code = Unknown desc = invalid bloom

Logs

1- The subnet archive validator node logs (truncated till that faulty blockID):

[02-15|07:17:19.701] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/block.go:289 built block {"blkID": "2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf", "innerBlkID": "NiAyL5yt9Uv13jjCoY1RpqTsTY3NHiRPMrQyQ4yVkuUkm6iyr", "height": 4362211, "parentTimestamp": "[02-14|06:14:03.000]", "blockTimestamp": "[02-15|07:17:19.000]"}
INFO [02-15|07:30:53.008] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/miner/worker.go:394: Commit new mining work number=4,362,218 hash=4a2781..681a21 uncles=0 txs=0 gas=0         fees=0            elapsed=1.677ms
[02-19|08:38:52.712] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:143 plugin handshake succeeded {"addr": "127.0.0.1:35733"}
[02-19|08:38:52.717] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:165 grpc: serving database {"address": "127.0.0.1:46439"}
[02-19|08:38:52.718] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:184 grpc: serving vm services {"address": "127.0.0.1:32801"}
INFO [02-19|08:38:52.723] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:291: Initializing Subnet EVM VM Version=v0.6.0-fuji Config="{AirdropFile: SnowmanAPIEnabled:false AdminAPIEnabled:false AdminAPIDir: WarpAPIEnabled:false EnabledEthAPIs:[eth eth-filter net web3 internal-eth internal-blockchain internal-transaction] ContinuousProfilerDir: ContinuousProfilerFrequency:15m0s ContinuousProfilerMaxFiles:5 RPCGasCap:50000000 RPCTxFeeCap:100 TrieCleanCache:512 TrieCleanJournal: TrieCleanRejournal:0s TrieDirtyCache:512 TrieDirtyCommitTarget:20 SnapshotCache:256 Preimages:false SnapshotWait:false SnapshotVerify:false Pruning:false AcceptorQueueLimit:64 CommitInterval:4096 AllowMissingTries:false PopulateMissingTries:<nil> PopulateMissingTriesParallelism:1024 PruneWarpDB:false MetricsExpensiveEnabled:true LocalTxsEnabled:false TxPoolJournal:transactions.rlp TxPoolRejournal:1h0m0s TxPoolPriceLimit:1 TxPoolPriceBump:10 TxPoolAccountSlots:16 TxPoolGlobalSlots:5120 TxPoolAccountQueue:64 TxPoolGlobalQueue:1024 APIMaxDuration:0s WSCPURefillRate:0s WSCPUMaxStored:0s MaxBlocksPerRequest:0 AllowUnfinalizedQueries:false AllowUnprotectedTxs:false AllowUnprotectedTxHashes:[0xfefb2da535e927b85fe68eb81cb2e4a5827c905f78381a01ef2322aa9b0aee8e] KeystoreDirectory: KeystoreExternalSigner: KeystoreInsecureUnlockAllowed:false RemoteGossipOnlyEnabled:false RegossipFrequency:1m0s RegossipMaxTxs:16 RegossipTxsPerAddress:1 PriorityRegossipFrequency:1s PriorityRegossipMaxTxs:32 PriorityRegossipTxsPerAddress:16 PriorityRegossipAddresses:[] LogLevel:info LogJSONFormat:false FeeRecipient: OfflinePruning:false OfflinePruningBloomFilterSize:512 OfflinePruningDataDirectory: MaxOutboundActiveRequests:16 MaxOutboundActiveCrossChainRequests:64 StateSyncEnabled:false StateSyncSkipResume:false StateSyncServerTrieCache:64 StateSyncIDs: StateSyncCommitInterval:16384 StateSyncMinBlocks:300000 StateSyncRequestSize:1024 InspectDatabase:false SkipUpgradeCheck:false AcceptedCacheSize:32 TxLookupLimit:0 WarpOffChainMessages:[]}"
INFO [02-19|08:38:52.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:443: Config has not specified any coinbase address. Defaulting to the blackhole address. 
INFO [02-19|08:38:52.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fd700 Disable:false}}"
INFO [02-19|08:38:52.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fd8b8 Disable:false} InitialMint:map[]}"
INFO [02-19|08:38:52.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fda70 Disable:false} InitialFeeConfig:<nil>}"
INFO [02-19|08:38:52.725] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="58.951µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [02-19|08:38:52.726] "<2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:457: lastAccepted = 0xe03d6b84c6086145327e7635a3e216bacf15f6a374445b8ba29281e6d70fb4b7" 
INFO [02-19|08:38:52.727] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:134: Allocated memory caches "trie clean"=512.00MiB "trie dirty"=512.00MiB "snapshot clean"=256.00MiB
INFO [02-19|08:38:52.733] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:171: Initialising Ethereum protocol network=17772 dbversion=8
INFO [02-19|08:38:52.742] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fd700 Disable:false}}"
INFO [02-19|08:38:52.742] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fd8b8 Disable:false} InitialMint:map[]}"
INFO [02-19|08:38:52.742] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fda70 Disable:false} InitialFeeConfig:<nil>}"
INFO [02-19|08:38:52.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="30.095µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
ERROR[02-19|08:38:52.750] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/genesis.go:246: found mismatch between config on database vs. new config storedConfig="{\"byzantiumBlock\":0,\"chainId\":17772,\"constantinopleBlock\":0,\"contractDeployerAllowListConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"feeConfig\":{\"gasLimit\":16000000,\"targetBlockRate\":2,\"minBaseFee\":24000000000,\"targetGas\":24000000,\"baseFeeChangeDenominator\":36,\"minBlockGasCost\":0,\"maxBlockGasCost\":10000000,\"blockGasCostStep\":500000},\"feeManagerConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"homesteadBlock\":0,\"istanbulBlock\":0,\"muirGlacierBlock\":0,\"petersburgBlock\":0,\"subnetEVMTimestamp\":0,\"upgrades\":{\"precompileUpgrades\":[{\"feeManagerConfig\":{\"blockTimestamp\":1671106500,\"disable\":true}},{\"contractNativeMinterConfig\":{\"blockTimestamp\":1671106500,\"disable\":true}},{\"feeManagerConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":1671112800}},{\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":1671112800}},{\"contractNativeMinterConfig\":{\"blockTimestamp\":1671127200,\"disable\":true}},{\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\",\"0x1a3624ec8355229cc4597c2746c92035cef26241\"],\"blockTimestamp\":1671186630}}]}}" newConfig="{\"byzantiumBlock\":0,\"chainId\":17772,\"constantinopleBlock\":0,\"contractDeployerAllowListConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"durangoTimestamp\":1707840000,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"feeConfig\":{\"gasLimit\":16000000,\"targetBlockRate\":2,\"minBaseFee\":24000000000,\"targetGas\":24000000,\"baseFeeChangeDenominator\":36,\"minBlockGasCost\":0,\"maxBlockGasCost\":10000000,\"blockGasCostStep\":500000},\"feeManagerConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"homesteadBlock\":0,\"istanbulBlock\":0,\"muirGlacierBlock\":0,\"petersburgBlock\":0,\"subnetEVMTimestamp\":0,\"upgrades\":{\"precompileUpgrades\":[{\"feeManagerConfig\":{\"blockTimestamp\":1671106500,\"disable\":true}},{\"contractNativeMinterConfig\":{\"blockTimestamp\":1671106500,\"disable\":true}},{\"feeManagerConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":1671112800}},{\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":1671112800}},{\"contractNativeMinterConfig\":{\"blockTimestamp\":1671127200,\"disable\":true}},{\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\",\"0x1a3624ec8355229cc4597c2746c92035cef26241\"],\"blockTimestamp\":1671186630}}]}}"
runtime engine: ignoring signal: terminated
runtime engine: ignoring signal: terminated
[02-19|12:57:19.072] ERROR <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/linux_stopper.go:58 subprocess was killed
[02-19|12:57:19.080] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:111 stdout collector shutdown
[02-19|12:57:19.080] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:124 stderr collector shutdown
[02-19|14:47:47.711] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:143 plugin handshake succeeded {"addr": "127.0.0.1:44835"}
[02-19|14:47:47.714] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:165 grpc: serving database {"address": "127.0.0.1:44867"}
[02-19|14:47:47.715] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:184 grpc: serving vm services {"address": "127.0.0.1:40081"}
INFO [02-19|14:47:47.717] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:291: Initializing Subnet EVM VM Version=v0.6.0-fuji Config="{AirdropFile: SnowmanAPIEnabled:false AdminAPIEnabled:false AdminAPIDir: WarpAPIEnabled:false EnabledEthAPIs:[eth eth-filter net web3 internal-eth internal-blockchain internal-transaction] ContinuousProfilerDir: ContinuousProfilerFrequency:15m0s ContinuousProfilerMaxFiles:5 RPCGasCap:50000000 RPCTxFeeCap:100 TrieCleanCache:512 TrieCleanJournal: TrieCleanRejournal:0s TrieDirtyCache:512 TrieDirtyCommitTarget:20 SnapshotCache:256 Preimages:false SnapshotWait:false SnapshotVerify:false Pruning:false AcceptorQueueLimit:64 CommitInterval:4096 AllowMissingTries:false PopulateMissingTries:<nil> PopulateMissingTriesParallelism:1024 PruneWarpDB:false MetricsExpensiveEnabled:true LocalTxsEnabled:false TxPoolJournal:transactions.rlp TxPoolRejournal:1h0m0s TxPoolPriceLimit:1 TxPoolPriceBump:10 TxPoolAccountSlots:16 TxPoolGlobalSlots:5120 TxPoolAccountQueue:64 TxPoolGlobalQueue:1024 APIMaxDuration:0s WSCPURefillRate:0s WSCPUMaxStored:0s MaxBlocksPerRequest:0 AllowUnfinalizedQueries:false AllowUnprotectedTxs:false AllowUnprotectedTxHashes:[0xfefb2da535e927b85fe68eb81cb2e4a5827c905f78381a01ef2322aa9b0aee8e] KeystoreDirectory: KeystoreExternalSigner: KeystoreInsecureUnlockAllowed:false RemoteGossipOnlyEnabled:false RegossipFrequency:1m0s RegossipMaxTxs:16 RegossipTxsPerAddress:1 PriorityRegossipFrequency:1s PriorityRegossipMaxTxs:32 PriorityRegossipTxsPerAddress:16 PriorityRegossipAddresses:[] LogLevel:info LogJSONFormat:false FeeRecipient: OfflinePruning:false OfflinePruningBloomFilterSize:512 OfflinePruningDataDirectory: MaxOutboundActiveRequests:16 MaxOutboundActiveCrossChainRequests:64 StateSyncEnabled:false StateSyncSkipResume:false StateSyncServerTrieCache:64 StateSyncIDs: StateSyncCommitInterval:16384 StateSyncMinBlocks:300000 StateSyncRequestSize:1024 InspectDatabase:false SkipUpgradeCheck:true AcceptedCacheSize:32 TxLookupLimit:0 WarpOffChainMessages:[]}"
INFO [02-19|14:47:47.719] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:443: Config has not specified any coinbase address. Defaulting to the blackhole address. 
INFO [02-19|14:47:47.719] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c1a0 Disable:false}}"
INFO [02-19|14:47:47.719] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c358 Disable:false} InitialMint:map[]}"
INFO [02-19|14:47:47.719] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c510 Disable:false} InitialFeeConfig:<nil>}"
INFO [02-19|14:47:47.720] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="46.085µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [02-19|14:47:47.722] "<2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:457: lastAccepted = 0xe03d6b84c6086145327e7635a3e216bacf15f6a374445b8ba29281e6d70fb4b7" 
INFO [02-19|14:47:47.723] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:134: Allocated memory caches "trie clean"=512.00MiB "trie dirty"=512.00MiB "snapshot clean"=256.00MiB
INFO [02-19|14:47:47.727] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:171: Initialising Ethereum protocol network=17772 dbversion=8
INFO [02-19|14:47:47.735] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c1a0 Disable:false}}"
INFO [02-19|14:47:47.736] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c358 Disable:false} InitialMint:map[]}"
INFO [02-19|14:47:47.736] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c510 Disable:false} InitialFeeConfig:<nil>}"
INFO [02-19|14:47:47.736] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="29.195µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [02-19|14:47:47.742] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/genesis.go:240: skipping verifying activated network upgrades on chain config 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:329:  
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:330: --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Chain ID:  17772 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Consensus: Dummy Consensus Engine 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Hard Forks: 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Mandatory Upgrades: 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - SubnetEVM Timestamp:           @0          (https://github.com/ava-labs/avalanchego/releases/tag/v1.10.0) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Durango Timestamp:            @1707840000 (https://github.com/ava-labs/avalanchego/releases/tag/v1.11.0) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Cancun Timestamp:              @nil        (https://github.com/ava-labs/avalanchego/releases/tag/v1.12.0) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Optional Network Upgrades: {} 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Precompile Upgrades: {"contractDeployerAllowListConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0},"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0},"feeManagerConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0}} 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Upgrade Config: {"precompileUpgrades":[{"feeManagerConfig":{"blockTimestamp":1671106500,"disable":true}},{"contractNativeMinterConfig":{"blockTimestamp":1671106500,"disable":true}},{"feeManagerConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":1671112800}},{"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":1671112800}},{"contractNativeMinterConfig":{"blockTimestamp":1671127200,"disable":true}},{"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7","0x1a3624ec8355229cc4597c2746c92035cef26241"],"blockTimestamp":1671186630}},{"warpConfig":{"blockTimestamp":1707850000,"quorumNumerator":0}}]} 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Fee Config: {"gasLimit":16000000,"targetBlockRate":2,"minBaseFee":24000000000,"targetGas":24000000,"baseFeeChangeDenominator":36,"minBlockGasCost":0,"maxBlockGasCost":10000000,"blockGasCostStep":500000} 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Allow Fee Recipients: false 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:334: --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  
INFO [02-19|14:47:47.749] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:732: Loaded most recent local header number=4,362,217 hash=e03d6b..0fb4b7 age=4d7h21m
INFO [02-19|14:47:47.749] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:733: Loaded most recent local full block number=4,362,217 hash=e03d6b..0fb4b7 age=4d7h21m
INFO [02-19|14:47:47.750] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1812: Loaded Acceptor tip hash=e03d6b..0fb4b7
INFO [02-19|14:47:47.750] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1820: Skipping state reprocessing root=505ec1..f880f2
INFO [02-19|14:47:47.751] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1788: Initializing snapshots async=true rebuild=true headHash=e03d6b..0fb4b7 headRoot=505ec1..f880f2
INFO [02-19|14:47:47.789] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:580: Warmed accepted caches start=4,362,186 end=4,362,217 t=37.361112ms
INFO [02-19|14:47:47.789] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:586: Starting Acceptor "queue length"=64
WARN [02-19|14:47:47.807] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2024-02-18T04:18:19-0500 age=1d10h29m
INFO [02-19|14:47:47.807] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/txpool/txpool.go:521: Transaction pool price threshold updated price=0
[02-19|14:47:47.812] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/vm.go:438 block height index was successfully verified
[02-19|14:47:47.812] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/vm.go:230 initialized proposervm {"state": "after fork", "forkHeight": 2, "lastAcceptedHeight": 4362217}
[02-19|14:47:47.813] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> snowman/transitive.go:111 initializing consensus engine
INFO [02-19|14:47:47.815] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:970: Enabled APIs: eth, eth-filter, net, web3, internal-eth, internal-blockchain, internal-transaction 
[02-19|14:47:47.817] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:159 starting bootstrapper
[02-19|14:47:49.920] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:312 bootstrapping started syncing {"numAccepted": 1}
[02-19|14:47:49.921] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:711 executing blocks {"numPendingJobs": 0}
[02-19|14:47:49.921] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> queue/jobs.go:224 executed operations {"numExecuted": 0}
[02-19|14:47:49.924] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> snowman/transitive.go:544 consensus starting {"lastAcceptedBlock": "2gRgasgqEaMczX5N3ov7U5ghpuCfKRa4pXGy1eGPNDBZLHB5jY"}
runtime engine: ignoring signal: terminated

2- The recent rpc node enters the network. The last line gives the FATAL error.

[03-04|12:37:25.567] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:143 plugin handshake succeeded {"addr": "127.0.0.1:45239"}
[03-04|12:37:25.570] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:160 grpc: serving database {"address": "127.0.0.1:37129"}
[03-04|12:37:25.570] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:179 grpc: serving vm services {"address": "127.0.0.1:44797"}
INFO [03-04|12:37:25.572] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:289: Initializing Subnet EVM VM Version=v0.6.2 Config="{AirdropFile: SnowmanAPIEnabled:true AdminAPIEnabled:false AdminAPIDir: WarpAPIEnabled:false EnabledEthAPIs:[public-eth public-eth-filter net web3 internal-public-eth internal-public-blockchain internal-public-transaction-pool internal-public-debug internal-private-debug internal-public-tx-pool internal-public-account debug-tracer] ContinuousProfilerDir: ContinuousProfilerFrequency:15m0s ContinuousProfilerMaxFiles:5 RPCGasCap:50000000 RPCTxFeeCap:100 TrieCleanCache:512 TrieCleanJournal: TrieCleanRejournal:0s TrieDirtyCache:512 TrieDirtyCommitTarget:20 TriePrefetcherParallelism:16 SnapshotCache:256 Preimages:false SnapshotWait:false SnapshotVerify:false Pruning:false AcceptorQueueLimit:64 CommitInterval:4096 AllowMissingTries:false PopulateMissingTries:<nil> PopulateMissingTriesParallelism:1024 PruneWarpDB:false MetricsExpensiveEnabled:true LocalTxsEnabled:false TxPoolPriceLimit:1 TxPoolPriceBump:10 TxPoolAccountSlots:16 TxPoolGlobalSlots:5120 TxPoolAccountQueue:64 TxPoolGlobalQueue:1024 TxPoolLifetime:10m0s APIMaxDuration:0s WSCPURefillRate:0s WSCPUMaxStored:0s MaxBlocksPerRequest:0 AllowUnfinalizedQueries:false AllowUnprotectedTxs:false AllowUnprotectedTxHashes:[0xfefb2da535e927b85fe68eb81cb2e4a5827c905f78381a01ef2322aa9b0aee8e] KeystoreDirectory: KeystoreExternalSigner: KeystoreInsecureUnlockAllowed:false PushGossipNumValidators:100 PushGossipNumPeers:0 PushRegossipNumValidators:10 PushRegossipNumPeers:0 PushGossipFrequency:100ms PullGossipFrequency:1s RegossipFrequency:30s PriorityRegossipAddresses:[] LogLevel:info LogJSONFormat:false FeeRecipient: OfflinePruning:false OfflinePruningBloomFilterSize:512 OfflinePruningDataDirectory: MaxOutboundActiveRequests:16 MaxOutboundActiveCrossChainRequests:64 StateSyncEnabled:false StateSyncSkipResume:false StateSyncServerTrieCache:64 StateSyncIDs: StateSyncCommitInterval:16384 StateSyncMinBlocks:300000 StateSyncRequestSize:1024 InspectDatabase:false SkipUpgradeCheck:true AcceptedCacheSize:32 TxLookupLimit:0 SkipTxIndexing:false WarpOffChainMessages:[]}"
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:442: Config has not specified any coinbase address. Defaulting to the blackhole address.
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3458 Disable:false}}"
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3678 Disable:false} InitialMint:map[]}"
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3888 Disable:false} InitialFeeConfig:<nil>}"
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="34.797µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [03-04|12:37:25.580] "<2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:456: lastAccepted = 0x45db17b560a3006acb419e645dea06b3560815cc4655eb1711b2293871907ca3"
INFO [03-04|12:37:25.581] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:142: Allocated memory caches "trie clean"=512.00MiB "trie dirty"=512.00MiB "snapshot clean"=256.00MiB
INFO [03-04|12:37:25.588] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:180: Initialising Ethereum protocol network=17772 dbversion=8
INFO [03-04|12:37:25.597] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3458 Disable:false}}"
INFO [03-04|12:37:25.597] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3678 Disable:false} InitialMint:map[]}"
INFO [03-04|12:37:25.597] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3888 Disable:false} InitialFeeConfig:<nil>}"
INFO [03-04|12:37:25.597] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="42.37µs"  gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [03-04|12:37:25.613] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/genesis.go:240: skipping verifying activated network upgrades on chain config
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:333: ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Chain ID:  17772
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Consensus: Dummy Consensus Engine
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Hard Forks:
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Mandatory Upgrades:
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - SubnetEVM Timestamp:           @0          (https://github.com/ava-labs/avalanchego/releases/tag/v1.10.0)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Durango Timestamp:            @1707840000 (https://github.com/ava-labs/avalanchego/releases/tag/v1.11.0)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Cancun Timestamp:              @nil        (https://github.com/ava-labs/avalanchego/releases/tag/v1.12.0)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Optional Network Upgrades: {}
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Precompile Upgrades: {"contractDeployerAllowListConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0},"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0},"feeManagerConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0}}
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Upgrade Config: {"precompileUpgrades":[{"feeManagerConfig":{"blockTimestamp":1671106500,"disable":true}},{"contractNativeMinterConfig":{"blockTimestamp":1671106500,"disable":true}},{"feeManagerConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":1671112800}},{"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":1671112800}},{"contractNativeMinterConfig":{"blockTimestamp":1671127200,"disable":true}},{"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7","0x1a3624ec8355229cc4597c2746c92035cef26241"],"blockTimestamp":1671186630}},{"warpConfig":{"blockTimestamp":1707850000,"quorumNumerator":0}},{"warpConfig":{"blockTimestamp":1707850001,"disable":true,"quorumNumerator":0}}]}
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Fee Config: {"gasLimit":16000000,"targetBlockRate":2,"minBaseFee":24000000000,"targetGas":24000000,"baseFeeChangeDenominator":36,"minBlockGasCost":0,"maxBlockGasCost":10000000,"blockGasCostStep":500000}
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Allow Fee Recipients: false
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:337: ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:338:
INFO [03-04|12:37:25.624] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:737: Loaded most recent local header number=4,362,210 hash=45db17..907ca3 age=2w5d1h
INFO [03-04|12:37:25.624] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:738: Loaded most recent local full block number=4,362,210 hash=45db17..907ca3 age=2w5d1h
INFO [03-04|12:37:25.628] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1818: Loaded Acceptor tip hash=45db17..907ca3
INFO [03-04|12:37:25.632] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1826: Skipping state reprocessing root=b1bf13..5f830f
INFO [03-04|12:37:25.632] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1794: Initializing snapshots async=true rebuild=true headHash=45db17..907ca3 headRoot=b1bf13..5f830f
INFO [03-04|12:37:25.691] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:585: Warmed accepted caches start=4,362,179 end=4,362,210 t=54.473813ms
INFO [03-04|12:37:25.691] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:591: Starting Acceptor "queue length"=64
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-06T22:04:34+0000 age=5mo4w1d
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-07T07:51:02+0000 age=5mo4w1d
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-07T14:37:03+0000 age=5mo4w22h
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-14T13:21:07+0000 age=5mo3w23h
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-22T09:52:53+0000 age=5mo2w2h
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-10-27T14:01:41+0000 age=4mo1w1d
INFO [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/txpool/txpool.go:523: Transaction pool price threshold updated price=0
[03-04|12:37:25.721] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/vm.go:436 block height index was successfully verified
[03-04|12:37:25.722] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/vm.go:228 initialized proposervm {"state": "after fork", "forkHeight": 2, "lastAcceptedHeight": 4362210}
[03-04|12:37:25.723] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> snowman/transitive.go:110 initializing consensus engine
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=public-eth new=eth
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=public-eth-filter new=eth-filter
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-eth new=internal-eth
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-blockchain new=internal-blockchain
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-transaction-pool new=internal-transaction
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-debug            new=internal-debug
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-private-debug           new=internal-debug
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-tx-pool          new=internal-tx-pool
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-account          new=internal-account
INFO [03-04|12:37:25.725] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:989: Enabled APIs: public-eth, public-eth-filter, net, web3, internal-public-eth, internal-public-blockchain, internal-public-transaction-pool, internal-public-debug, internal-private-debug, internal-public-tx-pool, internal-public-account, debug-tracer, snowman
[03-04|12:37:25.729] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:159 starting bootstrapper
[03-04|12:37:25.933] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:312 bootstrapping started syncing {"numAccepted": 1}
[03-04|12:37:27.092] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:711 executing blocks {"numPendingJobs": 91}
[03-04|12:37:27.129] ERROR <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/block_job.go:90 block failed verification during bootstrapping {"blkID": "2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf", "error": "rpc error: code = Unknown desc = invalid bloom (remote: 00000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000020000000000000000000800000000000000000000000000000000400000000082000000000000000000000000000000000000000000000000000004000000020000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000010000000000000000020000000000000000000000000000000000000000000000000000000000000000000  local: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)"}
[03-04|12:37:27.141] FATAL <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> handler/handler.go:337 shutting down chain {"reason": "received an unexpected error", "error": "failed to execute job 2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf due to failed to verify block in bootstrapping: rpc error: code = Unknown desc = invalid bloom (remote: 00000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000020000000000000000000800000000000000000000000000000000400000000082000000000000000000000000000000000000000000000000000004000000020000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000010000000000000000020000000000000000000000000000000000000000000000000000000000000000000  local: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000) while processing sync message: NodeID-6tWHHU9uVWAf46qTYTsnTnVZJeL65hwgR Op: ancestors Message: chain_id:\"\\xad\\xa9\\xf8K\\\\\\x0bH\\xbe\\x9d\\xaa\\xd8\\\\\\xf8F\\xb8\\xb8]lL<\\x99\\xd0R\\xca\\xf6s\\x9f\\xa2\\xfbb5\\xf5\"  request_id:2  containers:

Operating System Debian 11 x64

Any help or clarification will be appreciated

ceyonur commented 8 months ago

validators that run with the old code (<v0.6.0) after Durango has created a faulty state. Did you copy from a validator DB to new nodes? If so they also have this faulty state.

We're working on a way to mitigate missed upgrades.

thesoulkiller commented 7 months ago

we have 7 validators, 4 of them were operating normally but the other 3 fall in to this state. I copied db from one of the four validators to the others to mitigate the issue. I am looking forward for updates ! Thank you

ceyonur commented 7 months ago

Hello there! We just released Subnet-EVM v0.6.3 focusing on this very issue. You can now coordinate an upgrade for your network to reschedule the Durango activation. I advise to set a timestamp in future, giving enough time so your network can coordinate the upgrade. See here for more information. If you have any questions feel free to ask.

thesoulkiller commented 6 months ago

Hello I have upgraded to avalanchego-v1.11.4 and subnet-evm_0.6.3 but it couldn't detect networkUpgradeOverrides setting. I'm sharing the bootstrap logs.

Apr 22 07:41:28 vmi1540889.contaboserver.net avalanchego[1961877]: INFO [04-22|07:41:28.649] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:288: Initializing Subnet EVM VM Version=v0.6.3 Config="{AirdropFile: SnowmanAPIEnabled:false AdminAPIEnabled:false AdminAPIDir: WarpAPIEnabled:false EnabledEthAPIs:[eth eth-filter net web3 internal-eth internal-blockchain internal-transaction] ContinuousProfilerDir: ContinuousProfilerFrequency:15m0s ContinuousProfilerMaxFiles:5 RPCGasCap:50000000 RPCTxFeeCap:100 TrieCleanCache:512 TrieDirtyCache:512 TrieDirtyCommitTarget:20 TriePrefetcherParallelism:16 SnapshotCache:256 Preimages:false SnapshotWait:false SnapshotVerify:false Pruning:false AcceptorQueueLimit:64 CommitInterval:4096 AllowMissingTries:false PopulateMissingTries:<nil> PopulateMissingTriesParallelism:1024 PruneWarpDB:false MetricsExpensiveEnabled:true LocalTxsEnabled:false TxPoolPriceLimit:1 TxPoolPriceBump:10 TxPoolAccountSlots:16 TxPoolGlobalSlots:5120 TxPoolAccountQueue:64 TxPoolGlobalQueue:1024 TxPoolLifetime:10m0s APIMaxDuration:0s WSCPURefillRate:0s WSCPUMaxStored:0s MaxBlocksPerRequest:0 AllowUnfinalizedQueries:false AllowUnprotectedTxs:false AllowUnprotectedTxHashes:[0xfefb2da535e927b85fe68eb81cb2e4a5827c905f78381a01ef2322aa9b0aee8e] KeystoreDirectory: KeystoreExternalSigner: KeystoreInsecureUnlockAllowed:false PushGossipPercentStake:0.9 PushGossipNumValidators:100 PushGossipNumPeers:0 PushRegossipNumValidators:10 PushRegossipNumPeers:0 PushGossipFrequency:100ms PullGossipFrequency:1s RegossipFrequency:30s PriorityRegossipAddresses:[] LogLevel:info LogJSONFormat:false FeeRecipient: OfflinePruning:false OfflinePruningBloomFilterSize:512 OfflinePruningDataDirectory: MaxOutboundActiveRequests:16 MaxOutboundActiveCrossChainRequests:64 StateSyncEnabled:false StateSyncSkipResume:false StateSyncServerTrieCache:64 StateSyncIDs: StateSyncCommitInterval:16384 StateSyncMinBlocks:300000 StateSyncRequestSize:1024 InspectDatabase:false SkipUpgradeCheck:true AcceptedCacheSize:32 TxLookupLimit:0 SkipTxIndexing:false WarpOffChainMessages:[]}"
Apr 22 07:41:28 vmi1540889.contaboserver.net avalanchego[1961877]: [04-22|07:41:28.651] ERROR chains/manager.go:331 error creating chain {"subnetID": "21HEmZx5zVHYcP3JzbmRGVsYdm3HjrM2BMEPoCpoS3fHmZshq9", "chainID": "2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo", "chainAlias": "2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo", "vmID": "nyfSdZmrxTXbJrxdUoqLegVGQzWF6RVL4jYn7Yr6NsMzpdrFA", "error": "error while creating new snowman vm rpc error: code = Unknown desc = failed to parse upgrade bytes: unknown precompile config: networkUpgradeOverrides"}

EDIT My bad, I included the configuration in precompileUpgrades, when I just append the upgrade.json with this config, it worked. Thank you !